Remove 'memory/' from beginning of name for brevity

Blog — PlanetScale Posts about the PlanetScale platform, MySQL, PostgreSQL, databases, and more. https://planetscale.com/blog/feed.atom 2026-04-10T00:00:00.000Z Keeping a Postgres queue healthy https://planetscale.com/blog/keeping-a-postgres-queue-healthy 2026-04-10T00:00:00.000Z 2026-04-10T00:00:00.000Z Simeon Griggs Patterns for Postgres Traffic Control https://planetscale.com/blog/patterns-for-postgres-traffic-control 2026-04-02T00:00:00.000Z 2026-04-02T00:00:00.000Z Josh Brown Graceful degradation in Postgres https://planetscale.com/blog/graceful-degradation-in-postgres 2026-03-31T00:00:00.000Z 2026-03-31T00:00:00.000Z Ben Dicken High memory usage in Postgres is good, actually https://planetscale.com/blog/high-memory-usage-in-postgres-is-good-actually 2026-03-30T00:00:00.000Z 2026-03-30T00:00:00.000Z Simeon Griggs Stripe Projects partnership: Provision PlanetScale Postgres and MySQL databases from the Stripe CLI https://planetscale.com/blog/planetscale-stripe-projects-partnership 2026-03-26T00:00:00.000Z 2026-03-26T00:00:00.000Z Elom Gomez Enhanced tagging in Postgres Query Insights https://planetscale.com/blog/enhanced-tagging-in-postgres-query-insights 2026-03-24T00:00:00.000Z 2026-03-24T00:00:00.000Z Rafer Hazen Behind the scenes: How Database Traffic Control works https://planetscale.com/blog/behind-the-scenes-how-traffic-control-works 2026-03-23T16:00:00.000Z 2026-03-23T16:00:00.000Z Patrick Reynolds , and it matches any query that has that same value for that same key. It's complicated a bit by the fact that value can be an IP address with a CIDR mask. A rule set maps each pair to a rule. Now, when a query comes in with metadata like username=postgres, app=commerce, controller=api, the rule set can quickly identify the rule for each of those pairs. Hence, for this query, there are just three lookups in the rule set, regardless of how many rules are configured. Note that a rule set only identifies rules to consider. Each rule's budget is only checked if all its conditions match the query. A rule set is all about checking as few rules as possible. So, the sequence is: the rule set identifies a list of rules, that list is narrowed down to just the rules that actually match, and then the budgets for all the matching rules get checked to see if the query can proceed. There are three exceptions to the O(1) target for identifying rules: Rules for the remote_address key check for a match for each mask length. So if you have rules for ten different mask lengths, the rule set has to do as many as ten lookups to find the rule with the longest matching prefix. Any conjunction rule — that is, a rule with multiple pairs ANDed together — may be identified as a candidate for queries that match any one of the pairs in the rule. So if you have conjunction rules with overlapping pairs, the rule set may identify several or all of them as candidates for each query. It is possible to add multiple rules for the exact same pair. If you do that, any query with that exact pair will get checked against all of those rules. Applying new rules Traffic Control is meant to be used both proactively and during incident response. For incident response, it's important that rules take effect quickly. And they do! Rules created or modified in the UI generally take effect at all database replicas in just 1-2 seconds. How? Rules and budgets are stored as objects in the PlanetScale app. Any change to Traffic Control rules made in the UI or the API gets stored as rows in the planetscale database. Then it's serialized as JSON in the traffic_control.rules and traffic_control.budgets parameters for Postgres. Some Postgres parameters require restarting the server, but those two don't. So they cut the line and get sent immediately to postgresql.conf files on each database replica. Postgres reads the new config, and each worker process parses it into a rule set as soon as it completes whatever query it's executing. The rule set is in place before the next query begins. One big advantage of using Postgres configuration files, rather than sending configuration over SQL connections, is robustness on a busy server. You may want new Traffic Control rules most urgently when Postgres is using 100% of its available CPU, 100% of its worker processes, or both. Changing config files is possible even when opening a new SQL connection and issuing statements wouldn't be. Wrap up Traffic Control uses the hooks and the performance measurements that Query Insights already implemented, then bolts on a system for sorting query traffic into budgets and warning or blocking queries that exceed those budgets. Each query can be warned or blocked if it's individually too expensive, if too many other queries are already running under the same budget, or if recent and concurrent queries under the same budget have consumed too many resources in the aggregate. Traffic Control implements a dynamic model per query pattern that leverages the existing Postgres planner to estimate the real-world cost of a query before it begins to execute. Leaky buckets impose limits on both traffic bursts and the long-term average fraction of server resources assigned to any individual budget. Taken as a whole, these elements implement Traffic Control, which gives developers and database administrators powerful new tools to identify, prioritize, and limit SQL traffic.]]> Introducing Database Traffic Control https://planetscale.com/blog/introducing-database-traffic-control 2026-03-23T00:00:00.000Z 2026-03-23T00:00:00.000Z Sam Lambert Scaling Postgres connections with PgBouncer https://planetscale.com/blog/scaling-postgres-connections-with-pgbouncer 2026-03-13T00:00:00.000Z 2026-03-13T00:00:00.000Z Ben Dicken Postgres connections per pool).Since we have a clean user-to-logical-database mapping, we also set max_db_connections=2 and max_user_connections=2 to enforce this per-pool cap.The maximum total PgBouncer server connections is 200 × 2 = 400, matching max_connections=400. A single tenant can have 10s or even 100s of connections to PgBouncer, but all these will get multiplexed through at most 2 direct Postgres connections. App-side PgBouncers In some deployments, it also makes sense to layer PgBouncer.You can run one PgBouncer on the app or client side to funnel many worker or process connections into a smaller egress set, then run another PgBouncer near Postgres as the final funnel into a tightly controlled number of direct database connections. This is especially useful when you need connection pooling both close to compute and close to the database. Multiple PgBouncers In large-scale deployments, setting up multiple PgBouncers is useful for traffic isolation.When your web app, background workers, and other consumers all share one pool, a spike from one class of traffic can saturate the PgBouncer and delay everything else. Giving each major consumer its own PgBouncer creates independent funnels with their own limits, pool sizing, and failure domains.That makes it easier to protect latency-sensitive app traffic from bursty worker traffic and tune each workload separately. For an additional layer of protection, Database Traffic Control™ lets you enforce resource budgets on query traffic by pattern, application name, Postgres user, or custom tags — without needing separate infrastructure. The two approaches complement each other well: PgBouncer manages connections, Traffic Control manages resource consumption. The key concepts PgBouncer solves a fundamental architectural constraint in PostgreSQL: the process-per-connection model that makes every connection expensive.When working with PgBouncer, there are a few fundamental things to keep in mind: Transaction pooling is the mode that matters.Every transaction, be it a single query or many, gets a dedicated connection from PgBouncer <-> Postgres while executing.After this, the connection can be re-used for another transaction, maybe on the same client, and maybe for another. Use PgBouncer as much as possible.If you absolutely need features that are incompatible with transaction pooling, like LISTEN, session-level SET/RESET, or SQL PREPARE/DEALLOCATE, use a direct connection.In all other cases, the small latency penalty of PgBouncer is well worth the scalability and connection safety. The key configs to pay attention to are: max_connections (Postgres), plus max_client_conn, default_pool_size, max_db_connections, and max_user_connections (PgBouncer). Ensure things are configured to allow for direct connections, even when all PgBouncer connections are in use.]]> Drizzle joins PlanetScale https://planetscale.com/blog/drizzle-joins-planetscale 2026-03-03T00:00:00.000Z 2026-03-03T00:00:00.000Z Sam Lambert Video Conferencing with Postgres https://planetscale.com/blog/video-conferencing-with-postgres 2026-02-27T00:00:00.000Z 2026-02-27T00:00:00.000Z Nick Van Wiggeren Faster PlanetScale Postgres connections with Cloudflare Hyperdrive https://planetscale.com/blog/cloudflare-hyperdrive-real-time 2026-02-19T00:00:00.000Z 2026-02-19T00:00:00.000Z Simeon Griggs Introducing the PlanetScale MCP server https://planetscale.com/blog/introducing-planetscale-mcp-server 2026-01-29T00:00:00.000Z 2026-01-29T00:00:00.000Z Mike Coutermarsh Database Transactions https://planetscale.com/blog/database-transactions 2026-01-14T00:00:00.000Z 2026-01-14T00:00:00.000Z Ben Dicken Automating our changelog with Cursor commands https://planetscale.com/blog/automating-with-cursor-commands 2026-01-07T00:00:00.000Z 2026-01-07T00:00:00.000Z Mike Coutermarsh Postgres 18 is now available https://planetscale.com/blog/postgres-18-is-now-available 2025-12-17T00:00:00.000Z 2025-12-17T00:00:00.000Z Chris Sinjakli Using MotherDuck with PlanetScale https://planetscale.com/blog/using-motherduck-with-planetscale 2025-12-16T00:00:00.000Z 2025-12-16T00:00:00.000Z Ben Dicken $50 PlanetScale Metal is GA for Postgres https://planetscale.com/blog/50-dollar-planetscale-metal-is-ga-for-postgres 2025-12-15T00:00:00.000Z 2025-12-15T00:00:00.000Z Richard Crowley AI-Powered Postgres index suggestions https://planetscale.com/blog/postgres-new-index-suggestions 2025-11-21T00:00:00.000Z 2025-11-21T00:00:00.000Z Rafer Hazen $5 PlanetScale is live https://planetscale.com/blog/5-dollar-planetscale-is-here 2025-11-14T09:00:00.000Z 2025-11-14T09:00:00.000Z Sam Lambert Announcing Vitess 23 https://planetscale.com/blog/announcing-vitess-23 2025-11-04T00:00:00.000Z 2025-11-04T00:00:00.000Z Vitess Engineering Team $50 PlanetScale Metal https://planetscale.com/blog/50-dollar-planetscale-metal 2025-11-03T08:00:00.000Z 2025-11-03T08:00:00.000Z Sam Lambert Report on our investigation of the 2025-10-20 incident in AWS us-east-1 https://planetscale.com/blog/aws-us-east-1-incident-2025-10-20 2025-11-03T00:00:00.000Z 2025-11-03T00:00:00.000Z Richard Crowley $5 PlanetScale https://planetscale.com/blog/5-dollar-planetscale 2025-10-30T09:00:00.000Z 2025-10-30T09:00:00.000Z Sam Lambert Benchmarking Postgres 17 vs 18 https://planetscale.com/blog/benchmarking-postgres-17-vs-18 2025-10-14T00:00:00.000Z 2025-10-14T00:00:00.000Z Ben Dicken Larger than RAM Vector Indexes for Relational Databases https://planetscale.com/blog/larger-than-ram-vector-indexes-for-relational-databases 2025-10-01T00:00:00.000Z 2025-10-01T00:00:00.000Z Vicent Martí Partnering with Cloudflare to bring you the fastest globally distributed applications https://planetscale.com/blog/partnering-with-cloudflare-fastest-applications 2025-09-24T09:00:00.000Z 2025-09-24T09:00:00.000Z Mike Coutermarsh Processes and Threads https://planetscale.com/blog/processes-and-threads 2025-09-24T00:00:00.000Z 2025-09-24T00:00:00.000Z Ben Dicken PlanetScale for Postgres is now GA https://planetscale.com/blog/planetscale-for-postgres-is-generally-available 2025-09-22T00:00:00.000Z 2025-09-22T00:00:00.000Z Sam Lambert Postgres High Availability with CDC https://planetscale.com/blog/postgres-ha-with-cdc 2025-09-12T00:00:00.000Z 2025-09-12T00:00:00.000Z Sam Lambert Announcing Neki https://planetscale.com/blog/announcing-neki 2025-08-11T00:00:00.000Z 2025-08-11T00:00:00.000Z Andres Taylor Dirkjan Bussink Harshit Gangal Nick Van Wiggeren Noble Mittal Rohit Nayak Roman Sodermans Shlomi Noach Sam Lambert Caching https://planetscale.com/blog/caching 2025-07-08T00:00:00.000Z 2025-07-08T00:00:00.000Z Ben Dicken The principles of extreme fault tolerance https://planetscale.com/blog/the-principles-of-extreme-fault-tolerance 2025-07-03T09:00:00.000Z 2025-07-03T09:00:00.000Z Max Englander Announcing PlanetScale for Postgres https://planetscale.com/blog/planetscale-for-postgres 2025-07-01T09:00:00.000Z 2025-07-01T09:00:00.000Z Sam Lambert Postgres v13, as well as automatic Postgres version updates without downtime. Additionally, PlanetScale Metal’s locally-attached NVMe SSD drives fundamentally change the performance/cost ratio for hosting relational databases in the cloud. We’re excited to bring this performance to Postgres. Neki: Vitess for Postgres Vitess is one of PlanetScale’s greatest strengths and has become synonymous with database scaling. Contemporary Vitess is the product of PlanetScale’s experience running at extreme scale. We have made explicit sharding accessible to hundreds of thousands of users and it is time to bring this power to Postgres. We will not however be using Vitess to do this. Vitess’ achievements are enabled by leveraging MySQL’s strengths and engineering around its weaknesses. To achieve Vitess’ power for Postgres we are architecting from first principles. We are well under way with building this new system and will be releasing more information and early access as we progress. As with all PlanetScale products we work with customers at scale to build and validate maturity. If your company runs Postgres at a significant scale and this is something that interests you, reach out. You can also sign up for the Neki waitlist at neki.dev to stay updated on our progress. We are incredibly excited to be a part of the vibrant and thriving Postgres community.]]> Benchmarking Postgres https://planetscale.com/blog/benchmarking-postgres 2025-07-01T00:00:00.000Z 2025-07-28T00:00:00.000Z Ben Dicken Announcing Vitess 22 https://planetscale.com/blog/announcing-vitess-22 2025-04-29T00:00:00.000Z 2025-04-29T00:00:00.000Z Vitess Engineering Team PlanetScale vectors is now GA https://planetscale.com/blog/announcing-planetscale-vectors-ga 2025-03-25T12:00:00.000Z 2025-03-25T12:00:00.000Z Patrick Reynolds Faster interpreters in Go: Catching up with C++ https://planetscale.com/blog/faster-interpreters-in-go-catching-up-with-cpp 2025-03-20T00:00:00.000Z 2025-03-20T00:00:00.000Z Vicent Martí 100; Assuming this query is executed in a sharded Vitess cluster, the inventoried items can exist in any of the shards. Hence, our query planner will prepare a plan that queries all shards in parallel, pushing down part of the aggregation to MySQL, and then we'll perform the aggregations (SUM and AVG) locally in the vtgate. The state and warehouse checks in the WHERE clause can and will be executed directly on the MySQL instance that powers each shard. But the last expression, avg_price > 100, applies to the result of the aggregation, which is only available inside Vitess. This is where the Vitess evaluation engine comes in. Our evaluation engine is an interpreter that supports the majority of the scalar expressions in the SQL dialect used by MySQL. This does not include high level constructs such as performing a JOIN, the grouping of a GROUP BY, etc (these are performed directly by the planner, as we’ve seen), but the actual sub-expressions that you’d see as the condition of a WHERE clause, or a GROUP BY clause. Any piece of SQL that cannot be lowered to be executed in MySQL by the planner is evaluated locally in Go by the engine. Of course, these SQL sub-expressions are not arbitrarily complex. They are not even Turing complete (as they cannot loop!), so you may think that a statement like avg_price > ? would be trivial to evaluate, but as in most engineering problems, there’s a wealth of nuance when doing these things in the real world. SQL is an incredibly dynamic language full of quirks, and the SQL in MySQL, doubly so. We have spent an inordinate amount of time getting every single corner case of SQL evaluation to match exactly MySQL’s behavior. In fact, our test suite and fuzzer are so comprehensive that we routinely find bugs in the original MySQL evaluation engine, which we have to fix upstream (like this collation bug, this issue in the insert SQL function or this bug when searching substrings). Nonetheless, being fully accurate is not enough. For most queries, these expressions are evaluated once or even more than once for every returned row, so in order to not introduce additional overhead, evaluation needs to be as quick as possible. As discussed earlier, the first version of the evaluation engine in Vitess was an AST-based interpreter, operating directly on top of the SQL AST generated by our parser. This was a very straightforward design that allowed us to focus on accuracy, at the expense of performance. Let's discuss our new design for replacing this interpreter with a fully fledged virtual machine which is both faster and easier to maintain. Starting with the basics. The shapes of an interpreter For those new to programming language implementations, there are roughly 3 ways to execute a dynamic language at runtime. In increasing level of complexity and performance: An AST-based interpreter, where the syntax of the language is parsed into an AST and evaluation is performed by recursively walking each node of the AST and computing the results. (this is the way the evalengine in Vitess used to work!) A bytecode VM, where the AST is compiled into binary bytecode that can be evaluated by a virtual machine — a piece of code that simulates a CPU, but with higher-level instructions. (this is what we've recently shipped!) A JIT compiler, in which the bytecode is compiled directly into the host platform's native instructions, so it can be executed directly by the CPU without being interpreted by a Virtual Machine. (we'll talk about this later!) The first thing to consider here is whether the upgrade from an AST interpreter to a virtual machine makes sense from a performance point of view. Here’s an intuition: SQL expressions are incredibly dynamic (when it comes to typing), very high level (when it comes to each primitive operation), and with very little control flow (when it comes to evaluation -- SQL expressions don't really loop, and conditionals are rare; their flow is always lineal!). This can lead us to believe that there's no performance to be squeezed from translating the AST-based evaluation engine into bytecode. The AST is already well suited for high level operations and type-switching! This is only superficially true. Lots of programming languages are highly dynamic and they manage to run in bytecode VMs much more efficiently than with an AST interpreter. A clear example of this is the now ancient transition that Ruby did from its original AST interpreter in MRI to YARV, a bytecode VM. Python also did a similar switch very early on. And you can bet that literally no JavaScript engines are using AST evaluation: even though the goal of these engines is to start running JS as soon as possible, they still compile to (very efficient) bytecode interpreters before JIT compilation kicks in. So where’s the advantage of a virtual machine versus an AST interpreter? A lot of it boils down to instruction dispatching, which can be made very fast (more on this later!). But it is true that for SQL expressions, we’re actually going to execute very few instructions. Hence, to squeeze performance out of the VM, we’re going to have to come up with new tricks. The initial approach I had in mind for our SQL virtual machine was based on Efficient Interpretation using Quickening by Stefan Brunthaler. The idea behind this paper is that dynamic programming languages are very hard to execute efficiently because of the lack of information about types. A simple expression such as a + 1 must be interpreted in a completely different way depending on whether a is an integer, a floating point number of even a string. To optimize these operations in practice, the paper suggests rewriting the bytecode from more generic instructions (e.g. the sum operator that needs to figure out the types of the two operands to know how to sum them) into specific static instructions which are specialized for the types they operate on at runtime (e.g. the sum operator that knows that both operands are integers and can sum them right away). To do that, a quickening VM needs to figure out at runtime the types of the expressions being evaluated and incrementally rewrite the bytecode into instructions that operate on them. This is very hard to do in practice! But after implementing a good chunk of specialized instructions for the different types of operators in SQL and attempting to runtime rewrite them, I noticed an opportunity to take the idea even further by making it more efficient and, crucially, simpler. It turns out that the semantic analysis we perform in Vitess is advanced enough that, through careful integration with the upstream MySQL server and its information schema, it can be used to statically type the AST of the SQL expressions we were executing. This took a lot of effort to implement, but resulted in a big win: since the planner knows the types of the actual inputs that will be used to evaluate each SQL expression, we can derive from those the types of all sub-expressions at compilation time, resulting in byte-code that is already specialized without requiring runtime rewriting. Now we just need to implement a Virtual Machine to efficiently interpret the specialized bytecode! An efficient Virtual Machine in Go Implementing a VM usually involves a lot of complexity. As we’ve explained, you have to write a compiler that processes the input expression AST and generates the corresponding binary instructions (you have to come up with an encoding even!) and afterwards you have to implement the actual VM, which decodes each instruction and performs the corresponding operation. And you have to constantly keep these in sync! Any mismatches between the compiler that emits the bytecode and the VM that executes it are often catastrophic and very hard to debug. Historically, a bytecode VM has always been implemented the same way: a big-ass switch statement. You decode an instruction, and switch on its type to jump to the operation that needs to be performed. This is often a performance advantage against AST interpreters, because switching in practice is quite fast (particularly when implemented in C or C++ like most VMs are), and allows execution to happen linearly, without recursion. This design, however, also has its fair share of shortcomings. Mike Pall, JIT-master extraordinaire and author of LuaJIT, gives a very insightful rundown of these issues on this mailing list post from 2011. Allow me to summarize for this blog: Besides the fact that the VM's instructions need to be kept in-sync with the compiler, the actual performance of the main VM loop in a language with many instructions is not great in practice because compilers usually struggle when compiling massive functions, and these functions are massive. They spill registers all over the place on each branch of the switch, because it's hard to tell which branches are hot and which ones are cold. With all the pushing and popping, the jump into the switch's branch often looks more like a function call, so a lot of the performance benefits of the virtual machine dissipate. Mike was discussing C compilers in that post, but it's safe to assume that these problems are the same for a virtual machine implemented in Go. After a lot of testing, I can assure you that they are actually much worse because the Go compiler is not great at optimization. There’s always a trade-off between optimization and fast compile times, and the Go authors have historically opted for the latter. One key issue for Go is that often the different branches of the switch statement are jumped to via binary search instead of a jump table. Switch jump table optimization was implemented surprisingly late on the compiler, and in practice it is very fiddly, without any way to enforce it. You have to tweak the way the VM's instructions are encoded carefully to ensure that you're jumping in the VM's main loop, and you have no way to reliably check whether your virtual machine’s dispatch code has been properly optimized besides reviewing the generated assembly yourself. Clearly, switch-based VM loops are not the state of the art for writing efficient interpreters, neither in Go nor in any other programming language. So what is the state of the art then? Well, when it comes to Go it turns out that there's nobody doing fast interpreters right now (at least nobody I can find). The people who are doing interesting work here, such as the wazero WASM implementation are focusing their performance efforts on JIT. So we’re going to have to innovate! Outside of Go, the most interesting approach for interpreters implemented in C or C++ is continuation-style evaluation loops, as seen in this report from 2021 that implements this technique for parsing Protocol Buffers. This involves implementing all the opcodes for the VM as freestanding functions that operate on the VM as an argument, with the return of the function being a callback to the next step of the computation. It does sound like something expensive and, huh, recursive, but the trick is that newer versions of LLVM allow us to mark functions as forcefully tail-called (see: https://en.wikipedia.org/wiki/Tail_call), so the resulting code is not recursively calling the VM loop but instead jumping between the operations and using the free-standing functions as an abstraction to control register placement and spillage. The most recent release of Python 3.14 actually ships an interpreter based on this design, boasting up to 30% improvement when executing Python code. Unfortunately, this is not something we can do in Go because as we discussed earlier, the Go compiler is allergic to optimization. It can sometimes emit tail calls, but it needs to be tickled in just the right way, and this implementation simply does not work in practice unless the tail-calls are guaranteed at compilation time. But what if we keep the same design with free-standing functions for each instruction and instead of tail-calling, we forcefully return control to the evaluation loop after each one? This could be implemented very easily by not emitting our compiled program as “byte code”, but instead emitting a slice of function pointers to each instruction. The design may be a bit counter-intuitive, but it has a lot of very interesting properties. First, the VM becomes trivial! It's just a few lines of code, and it doesn't have to worry about optimizing any large switch statements. It's just repeatedly calling functions one after the other! Here’s a simplified example, but if you check the actual implementation in Vitess you’ll see that a real virtual machine implementation is hardly more complicated than this.func (vm *VirtualMachine) execute(p *Program) (eval, error) { code := p.code ip := 0 for ip < len(code) { ip += code[ip](vm) if vm.err != nil { return nil, vm.err } } if vm.sp == 0 { return nil, nil } return vm.stack[vm.sp-1], nil } All we need to return when executing each instruction is the offset for the instruction pointer ip. Most functions return 1, which causes the next instruction to be executed, but by returning negative or positive values, you can implement all control flow, including loops and conditionals. Besides the greatly simplified virtual machine, the second advantage of this approach is that the compiler also becomes trivial, because there is no bytecode! Instead, the compiler emits the individual instructions directly by pushing "callbacks" into a slice. There are no instruction opcodes to keep track off, no encoding to perform and nothing to keep in sync with the VM. Developing the compiler means developing the VM simultaneously, which greatly improves iteration speed and prevents a whole class of bugs that happen often when developing virtual machines.func (c *compiler) emitPushNull() { c.emit(func(vm *VirtualMachine) int { vm.stack[vm.sp] = nil vm.sp++ return 1 }) } As you may notice, there’s a bit of a hiccup here when it comes to modeling the instructions for a non-trivial language: if there's no instruction encoding, then we cannot have instructions with arguments. This is a big problem in a language like C (traditionally used to implement most programming language interpreters), which is why this technique is never seen there. But it’s actually not a problem for us, because the Go compiler actually supports closures! We can emit any instruction we want and the Go compiler will automatically capture its arguments inside the callback. We don't have to think about how to encode our arguments in the bytecode, and in fact, our arguments can be as complex as they need to be: the resulting callback will contain a copy of them created by the Go compiler. It's essentially a poor man's JIT, aided by the compiler, and it works amazingly well in practice, both performance-wise and for ergonomics. Check out this compiler method that generates an instruction to push a TEXT SQL object from the input rows into the stack:func (c *compiler) emitPushColumn_text(offset int, col collations.TypedCollation) { c.emit(func(vm *VirtualMachine) int { vm.stack[vm.sp] = newEvalText(vm.row[offset].Raw(), col) vm.sp++ return 1 }) } Both the offset in the input rows array and the collation for the text are statically baked into the generated instruction! Almost statically typed With the fully static typing for SQL expressions (derived from the type information in the planner) we get to design an extremely efficient virtual machine where every single instruction is specialized for the type of the operands it executes on. This is both the optimal and the simplest design for a VM because we never have to do type switching during evaluation. But we’re dealing with SQL here (or, more accurately, the SQL dialect of MySQL), so not everything is rainbows and unicorns. Very often it’s quite the opposite. Let’s consider this wildly complex SQL expression: -inventory.price. That is, the negation of each of the values in the inventory.price column of our query. We know (thanks to our semantic analysis, and the schema tracker) that the type of the inventory.price column is BIGINT. So what could be the type of -inventory.price? Naive readers without experience in the magical world of SQL may believe the resulting type is BIGINT, but that’s not the case in practice! The vast majority of the time, the negation of a BIGINT yields indeed another BIGINT value. But when the actual value of the BIGINT is -9223372036854775808 (i.e. the smallest value that can be represented in 64 bits), negating it promotes the value into a DECIMAL, instead of silently truncating it, or returning an error. You can see how this can easily throw a wrench in our statically compiled instructions for our virtual machine. Suddenly the static type checking we’ve computed is no longer valid because the types of the expression no longer depend on the types of the inputs, but on the actual values of the inputs. In order to continue evaluating the result of this negation, we’d always have to type-check again at runtime, defeating the whole point of static typing to begin with. To work around this issue, we’re not introducing more type switches at runtime. We’re using a classic trick which can be seen all the time in JIT compiled code and very rarely, if ever, in virtual machines: de-optimization. There’s a small list of expressions where corner cases (e.g. overflow) can result in dynamic typing at runtime. Whenever this happens, we simply bail out of executing in our virtual machine and fall back to executing on the old AST evaluator, which has always performed type switching at runtime. This is very similar to what JIT compilers do when they detect that the runtime type of a value no longer matches the generated code they’ve emitted; they fall back from the native code to the virtual machine. In our case, we’re one step behind, falling back from the virtual machine to the AST interpreter, but the performance implications are the same. This design allows us to keep our interpreter executing statically typed code without any type switches at runtime. Here's an example of what integer negation looks like when compiled:func (c *compiler) emitNeg_i() { c.emit(func(vm *VirtualMachine) int { arg := vm.stack[env.vm.sp-1].(*evalInt64) if arg.i == math.MinInt64 { vm.err = errDeoptimize } else { arg.i = -arg.i } return 1 }) } There is one significant drawback with this approach, however: the code for the AST interpreter can never be removed from Vitess. But this is, overall, not a bad thing. Just like most advanced language runtimes keep their virtual machine interpreter despite having a JIT compiler, having access to our classic AST interpreter gives us versatility. It can be used when we detect that an expression will be evaluated just once (e.g. when we use the evaluation engine to perform constant folding on a SQL expression). In those cases, the overhead of compiling and then executing on the VM trumps a single-pass evaluation on the AST. Lastly, when it comes to accuracy, being able to fuzz both the AST interpreter and the VM against each other has resulted in an invaluable tool for detecting bugs and corner cases. Conclusion This technique for virtual machine implementation is not fully novel (I’ve seen it used before for a rules-based authorization engine in the wild!), but as far as I can tell it has never been used in Go. Given the constraints of the language and the compiler, the technique yields spectacular results: the new SQL interpreter in Vitess is just faster. Faster to write, faster to maintain and faster to execute. The benchmarks speak for themselves: Evalengine performance in Vitess over time Here we have a performance comparison of 5 different queries (ranging from very complex to very simple) between three implementations: old, which is the original AST-based dynamic implementation of the evalengine. ast, which is the result of adding static type checking to the virtual machine and using them to partially optimize the AST evaluator. vm, which is the callback-based virtual machine implementation as discussed in this post. Recent results compared with MySQL This is the current performance of our evaluation engine pitted against the native C++ implementation in MySQL. Note that measuring the time that MySQL spends in evaluation is very tricky; these are not the total reponse times for a query, but the result of manual instrumentation in the mysqld server to ensure a fair comparison. │ ast │ vm │ mysql │ │ sec/op │ sec/op vs base │ sec/op vs base │ CompilerExpressions/complex_arith-32 162.75n ± 1% 50.77n ± 1% -68.81% (p=0.000 n=10) 49.40n ± 5% -69.64% (p=0.000 n=10+184) CompilerExpressions/comparison_i64-32 30.30n ± 2% 16.95n ± 1% -44.08% (p=0.000 n=10) 26.93n ± 22% -11.12% (p=0.000 n=10+11) CompilerExpressions/comparison_u64-32 30.57n ± 3% 17.49n ± 1% -42.78% (p=0.000 n=10) 18.80n ± 9% -38.53% (p=0.000 n=10+16) CompilerExpressions/comparison_dec-32 70.75n ± 1% 52.58n ± 2% -25.68% (p=0.000 n=10) 46.59n ± 5% -34.14% (p=0.000 n=10+14) CompilerExpressions/comparison_f-32 53.05n ± 1% 25.65n ± 1% -51.64% (p=0.000 n=10) 27.75n ± 23% -47.69% (p=0.000 n=10) geomean 56.30n 28.94n -48.60% 31.76n -43.58% │ ast │ vm │ │ B/op │ B/op vs base │ CompilerExpressions/complex_arith-32 96.00 ± 0% 0.00 ± 0% -100.00% (p=0.000 n=10) CompilerExpressions/comparison_i64-32 16.00 ± 0% 0.00 ± 0% -100.00% (p=0.000 n=10) CompilerExpressions/comparison_u64-32 16.00 ± 0% 0.00 ± 0% -100.00% (p=0.000 n=10) CompilerExpressions/comparison_dec-32 64.00 ± 0% 40.00 ± 0% -37.50% (p=0.000 n=10) CompilerExpressions/comparison_f-32 16.00 ± 0% 0.00 ± 0% -100.00% (p=0.000 n=10) │ ast │ vm │ │ allocs/op │ allocs/op vs base │ CompilerExpressions/complex_arith-32 9.000 ± 0% 0.000 ± 0% -100.00% (p=0.000 n=10) CompilerExpressions/comparison_i64-32 1.000 ± 0% 0.000 ± 0% -100.00% (p=0.000 n=10) CompilerExpressions/comparison_u64-32 1.000 ± 0% 0.000 ± 0% -100.00% (p=0.000 n=10) CompilerExpressions/comparison_dec-32 3.000 ± 0% 2.000 ± 0% -33.33% (p=0.000 n=10) CompilerExpressions/comparison_f-32 2.000 ± 0% 0.000 ± 0% -100.00% (p=0.000 n=10) The results are stark: the pre-compiled SQL expressions when ran in the new VM are up to 20x times faster than the first implementation of SQL evaluation in Vitess, and for most cases, we've caught up with the performance of the C++ implementation in MySQL. One further detail which is not shown on the graphs, but can be seen on the raw benchmark data, is that the new virtual machine does not allocate memory to perform evaluation — a very nice side effect of the fully specialized instructions thanks to the static type checking. Overall, we consider getting in the same performance ballpark as MySQL's C++ evaluation engine as a huge engineering success, particularly when the resulting implementation is so easy to maintain.There will always be a performance gap between Go and C++, arising from the trade-off of quality vs compilation speed in the Go compiler, and from the semantics of the language itself, but as we show here, this gap is not insurmountable. With expertise and careful design, it is possible to reap the many benefits of developing and deploying Go services without paying the performance penalty inherent in the language. In this specific case, we got there by having the capacity to perform semantic analysis and statically typing SQL expressions (something which MySQL does not do), and by choosing an efficient virtual machine design that uses the strengths of Go instead of fighting its limitations. Addendum: So why not JIT? Inquiring minds may be wondering: what's next? Are we doing JIT compilation next? The answer is no. Although this design for a compiler and VM looks like an exceptional starting point for implementing a full JIT compiler in theory, in practice the trade-off between optimization and complexity doesn't make sense. JIT compilers are important for programming languages where their bytecode operations can be optimized into a very low level of abstraction (e.g. where an "add" operator only has to perform a native x64 ADD). In these cases, the overhead of dispatching instructions becomes so dominant that replacing the VM's loop with a block of JITted code makes a significant performance difference. However, for SQL expressions, and even after our specialization pass, most of the operations remain extremely high level (things like "match this JSON object with a path" or "add two fixed-width decimals together"). The overhead of instruction dispatch, as measured in our benchmarks, is less than 20% (and can possibly be optimized further in the VM's loop). 20% is not the number you're targetting before you start messing around with raw assembly for a JIT. So at this point my intuition is that JIT compilation would be a needlessly complex dead optimization.]]> The Real Failure Rate of EBS https://planetscale.com/blog/the-real-fail-rate-of-ebs 2025-03-18T00:00:00.000Z 2025-03-18T00:00:00.000Z Nick Van Wiggeren IO devices and latency https://planetscale.com/blog/io-devices-and-latency 2025-03-13T00:00:00.000Z 2025-03-13T00:00:00.000Z Ben Dicken Announcing PlanetScale Metal https://planetscale.com/blog/announcing-metal 2025-03-11T00:00:00.000Z 2025-03-11T00:00:00.000Z Sam Lambert PlanetScale Metal: There’s no replacement for displacement https://planetscale.com/blog/planetscale-metal-theres-no-replacement-for-displacement 2025-03-11T00:00:00.000Z 2025-03-11T00:00:00.000Z Richard Crowley Upgrading Query Insights to Metal https://planetscale.com/blog/upgrading-query-insights-to-metal 2025-03-11T00:00:00.000Z 2025-03-11T00:00:00.000Z Rafer Hazen Automating cherry-picks between OSS and private forks https://planetscale.com/blog/automating-cherry-picks-between-oss-and-private-forks 2025-01-14T00:00:00.000Z 2025-01-14T00:00:00.000Z Manan Gupta Database Sharding https://planetscale.com/blog/database-sharding 2025-01-09T00:00:00.000Z 2025-01-09T00:00:00.000Z Ben Dicken Anatomy of a Throttler, part 3 https://planetscale.com/blog/anatomy-of-a-throttler-part-3 2024-11-19T00:00:00.000Z 2024-11-19T00:00:00.000Z Shlomi Noach Introducing sharding on PlanetScale with workflows https://planetscale.com/blog/introducing-workflows-on-planetscale 2024-11-07T10:00:00.000Z 2024-11-07T10:00:00.000Z Ben Dicken Announcing Vitess 21 https://planetscale.com/blog/announcing-vitess-21 2024-10-29T09:01:00.000Z 2024-10-29T09:01:00.000Z Vitess Engineering Team Announcing the PlanetScale vectors public beta https://planetscale.com/blog/announcing-planetscale-vectors-public-beta 2024-10-21T10:00:00.000Z 2024-10-21T10:00:00.000Z Holly Guevara Anatomy of a Throttler, part 2 https://planetscale.com/blog/anatomy-of-a-throttler-part-2 2024-10-10T00:00:00.000Z 2024-10-10T00:00:00.000Z Shlomi Noach B-trees and database indexes https://planetscale.com/blog/btrees-and-database-indexes 2024-09-09T00:00:00.000Z 2024-09-09T00:00:00.000Z Ben Dicken $START_DATETIME AND sent < $END_DATETIME ORDER BY sent DESC; Consider what this would be like if we have UUIDv4s for our primary key.In the B+tree below, a bunch of random keys and corresponding values have been inserted into the table.Try finding ranges of values.What do you see? Notice that the value sequences are spread out across many non-sequential leaf nodes.On the other hand, consider finding sequentially inserted values instead. In such cases, all pages with the search results will be next to each other.It's even possible to search for several rows, and all of them will be next to each other in a single page.For this variety of query pattern, we can mitigate the number of pages that need to be read using a sequential primary key. Primary key size Another important consideration is key size.We always want our primary keys to be: Big enough to never face exhaustion Small enough to not use excessive storage For integer sequences, we can sometimes get away with a MEDIUMINT (16 million unique values) or INT (4 billion unique values) for smaller tables.For big tables, we often jump to BIGINT to be safe (18 sextillion possible values).BIGINTs are 64 bits (8 bytes).UUIDs are typically 128 bits (16 bytes), twice the size of even the largest integer type in MySQL.Since B+tree nodes are a fixed size, a BIGINT will allow us to fit more keys per-node than UUIDs.This results in shallower trees and faster lookups. Consider a case where each tree node is only 100 bytes, child pointers are 8 bytes, and values are 8 bytes.We could fit 4 UUIDs (plus 4 child pointers) in each node.Hit the play insertion sequence button below to see the inserts. If we had used a BIGINT instead, we could fit 6 keys (and corresponding child pointers) in each node instead.This would lead to a shallower tree, better for performance. Pages and InnoDB Recall that one of the big benefits of a B+tree is the fact that we can set the node size to whatever we want.In InnoDB, the B+tree nodes are typically set to 16k, the size of an InnoDB page. When fulfilling a query (and therefore traversing B+trees), InnoDB does not read individual rows and columns from disk.Whenever it needs to access a piece of data, it loads the entire associated page from disk. InnoDB has some tricks up its sleeve to mitigate this, the main one being the buffer pool.The buffer pool is an in-memory cache for InnoDB pages, sitting between the pages on-disk and MySQL query execution.When MySQL needs to read a page, it first checks if it's already in the buffer pool.If so, it reads it from there, skipping the disk I/O operation.If not, it finds the page on-disk, adds it to the buffer pool, and then continues query execution. The buffer pool drastically helps query performance.Without it, we'd end up doing significantly more disk I/O operations to handle a query workload.Even with the buffer pool, minimizing the number of pages that need to be visited helps performance (1) because there's still a (small) cost to looking up a page in the buffer pool, and (2) it helps reduce the number of buffer pool loads and evictions that need to take place. Other situations Here, we mostly focused on comparing a sequential key to a random / UUID key.However, the principles shown here are useful to keep in mind no matter what kind of primary or secondary key you are considering. For example, you may also consider using a user.created_at timestamp as a key for an index.This will have similar properties to a sequential integer.Insertions will generally always go to the right-most path, unless legacy data is being inserted. Conversely, something like a user.email_address string will have more similar characteristics to a random key.Users won't be creating accounts in email-alphabetical order, so insertions will happen all over the place in the B+tree. Conclusion This is already a long blog post, and yet, much more could be said about B+trees, indexes, and primary key choice in MySQL.On the surface it may seem simple, but there's an incredible amount of nuance to consider if you want to squeeze every ounce of performance out of your database.If you'd like to experiment further, you can visit the dedicated interactive B+tree website.If you want a regular B-tree, go here instead.I hope you learned a thing or two about indexes! Special thanks to Sam Rose for early review.]]> Instant deploy requests https://planetscale.com/blog/instant-deploy-requests 2024-09-04T16:01:00.000Z 2024-09-04T16:01:00.000Z Shlomi Noach Anatomy of a Throttler, part 1 https://planetscale.com/blog/anatomy-of-a-throttler-part-1 2024-08-29T00:00:00.000Z 2024-08-29T00:00:00.000Z Shlomi Noach Increase IOPS and throughput with sharding https://planetscale.com/blog/increase-iops-and-throughput-with-sharding 2024-08-19T00:00:00.000Z 2024-08-19T00:00:00.000Z Ben Dicken Tracking index usage with Insights https://planetscale.com/blog/tracking-index-usage-with-insights 2024-08-14T00:00:00.000Z 2024-08-14T00:00:00.000Z Rafer Hazen 1000 p50:>250. To learn more about how and why we implemented this feature, read on. Existing tools and implementation Before building a new system to monitor index usage, we evaluated the available tools for monitoring and understanding index usage. We’ll explore these tools first to motivate the decisions we made in designing Insights usage tracking. The first tool most developers reach for when trying to understand index usage is explain, and for good reason. Explain is an incredibly powerful tool that exposes a wealth of information about how MySQL executes your query, including index usage. It’s great for troubleshooting a problematic query or testing out a new index. Unfortunately, explain can only provide information for queries you explicitly provide. It doesn’t record information for the actual queries processed by MySQL, so it can’t show how a query pattern is using indexes over time, across shards, or with the different query parameters from your production workload. For aggregate index usage from your production environment, MySQL’s built in performance schema provides counters for how many times each index has been used in the table_io_waits_summary_by_index_usage table. This is a useful facility, but comes with a number of limitations that make it difficult to use in practice: Stats are in the form of global counts for each MySQL server, and are reset when MySQL is restarted. This means you can’t see usage trends over time, and counts may be reset at any time. Counters are only provided at the index level, so it’s not clear which query patterns are using which indexes. To make it easy to understand index usage patterns, we wanted a system that: Breaks down index usage information per query pattern. Stores index usage as a timeseries, so it’s obvious when something has changed. Provides cumulative data for all your queries, without sampling or extrapolating based on explain plans. With these goals in mind, our first task was to extract query index usage information from MySQL. Since PlanetScale databases exclusively use the InnoDB storage engine, we were able to focus our efforts there. The InnoDB storage handler includes an index-initialization function that MySQL calls (once) prior to using an index in a query. By recording the index name passed to this function in a per query data structure, we’re able to find the set of all indexes used by each query. When the query is finished we return the list of used indexes in the final packet returned by MySQL to the client, and ultimately to VTGate, Vitess’s query proxying layer. With the per-query index information in VTGate, we aggregate index usage information per query-pattern and send it into the Insights pipeline every 15 seconds. This approach allows us to aggregate the time series count of indexes used for 100% of queries with negligible overhead in MySQL. Try it out now Index usage information is available on all PlanetScale databases. We’ve found this feature useful in managing our own databases and we hope you do too.]]> Zero downtime migrations at petabyte scale https://planetscale.com/blog/zero-downtime-migrations-at-petabyte-scale 2024-08-13T00:00:00.000Z 2024-08-13T00:00:00.000Z Matt Lord PlanetScale->OldDB with the query response following that same path in reverse. This will incur a significant performance cost — all the more so if the application was previously connecting to the original database using a fast local connection (unix domain socket, loopback device, etc). It's for this same reason that you should not attempt to do application side performance comparisions using PlanetScale, comparing it to the usage of your original database, until you've done the cutover. This is because in this temporary pre-cutover stage the bulk of the total execution time will be spent in network round trips for well optimized queries. Please note, however, that when using PlanetScale Managed — as most customers at this scale would be doing — the additional network cost would be minimal as the application and PlanetScale database would typically be in the same cloud vendor account and sharing the same physical locations (regions and availability zones) and physical networks in those locations. You can remain in this state as long as necessary for you to prepare for the application cutover and perform additional testing of the application and database system. Transparently cutover application traffic to the new system so that application traffic is going to PlanetScale. During this process we ensure that there is no data loss or drift. Incoming queries are paused/buffered while performing the system changes for the traffic switch. Once the cutover is complete — which would typically take less than 1 second — the paused/buffered queries are executed and the system is back to normal operation. Reverse replication is put in place so that if for any reason we need to revert the cutover, we can do so without data loss or downtime (this can be done back and forth as many times as necessary). The query buffering done here is the last part that allows the entire migration to be done without any downtime. Whenever you are confident in the system and no longer require the need to cut traffic back over to your old system, you can finish or complete the migration. This is the final step in the migration process and is the point at which you can decommission your old system whenever you like. At no point during this process is the old system down or are application users aware that anything out of the ordinary is occurring. The only thing one would typically notice is the slight spike in query latencyat the point of cutover where we briefly pause the incoming queries so that we can route them to the correct side of the migration (old vs new) once the cutover work is done. When you've reached a certain scale — where a database is larger than 250GiB being the recommendation in Vitess — horizontally sharding your databaseis the best way to continue scaling without incurring exponentially more expensive hardware related costs and affecting the performance of queries and various operations (e.g. schema changes and backups). This is a key feature of Vitess and PlanetScale and it is typical to shard adatabase as part of the data migration. So e.g. you can have an unsharded MySQL database that we then split into N shards as part of the data migration into PlanetScale. In fact, being able to do this is a common reason for migrating to PlanetScale in the first place. A deep dive into the technical details PlanetScale is a database-as-a-service offering a "modern MySQL" developer experience, built around Vitess. Vitess offers a suite of related tools and primitives related to data migrationscalled VReplication. The PlanetScale feature built around that set of Vitess primitives for imports is called (not surprisingly) Database Imports. This feature provides the web and CLI based user interfaces aroundVReplication's MoveTables workflow for the consistent data copy (streaming rows) and the replication (streaming binary log events). If we walk through the process of a data migration at PlanetScale again at a lower level (the steps are somewhat simplified here with certain details and optional behaviors left out, but it covers the main points): Copy the existing data by taking a consistent non-locking snapshot of the data which includes the metadata needed to replicate changes that happen after our snapshot was taken — which we will be doing throughout the migration process. a. Each of the N shards in the PlanetScale database cluster has a PRIMARY tablet. Those connect to the unmanaged tablet that is placed in front of theexternal MySQL instance (the old system) and initiates a stream where we issue a LOCK TABLE READ query to make it read-only just long enough to issue START TRANSACTION WITH CONSISTENT SNAPSHOTand read the @@global.GTID_EXECUTED value, then releasing the lock. At this point we have a consistent snapshot of the table data and the GTID set or "position"metadata to go along with it so that we can replicate changes to the table that have occurred since our snapshot was taken. b. We then start reading all of the rows from our snapshot — at a logical point in time — ordering the results by the PRIMARY KEY (PK) columns in the table (if there are none then we will use the best PK equivalent, meaning non-null unique key) so that we can read from theclustered index immediately as we are then reading the records in order and do not need to formulate the entire result set and order it with a filesort before we canstart streaming rows. The source tablet then has N streams, where each stream is going to a target shard's PRIMARY tablet, and each stream filters out any rows from our query results that are not going to the stream'starget shard based on the sharding scheme defined for the table. The applicable rows are then sent to the target shard's PRIMARY tablet where they are inserted into the table and metadata for the stream is updated there(in the sidecar database's vreplication and copy_state tables) so that we can continue where we leftoff when the copy is interrupted for any reason. The streams continue to copy rows until we've completed copying all of the rows in our snapshot or we hit the configured copy phase cycle duration(see Life of a Stream for more details) — in which case we will pause the row copy work and catch up on all of the changes that have happened to the rows we've copied so far by streaming the applicable binary log events. The binlogevents are filtered in each stream by the destination shard and whether or not the change is applicable to a row we've copied as otherwise we'll get a later version of the row in a subsequent copy phase cycle. This regular catchup step is important to ensure that we don't completethe row copy only to then be unable to replicate from where we left off because the source MySQL instance no longer has binary log events that we need as they have been purged — in which case we would be forced to start the entiremigration over again. This also happens to improve the performance of the overall operation as we are replicating the minimal events needed to ensure eventual consistency. c. We do this table by table (serially), across all of the streams (the streams running concurrently), until we're done copying the initial table data. As you can imagine, executing all of this work on your current live database instance can be somewhat heavy or expensive and potentially interfere with your live application traffic and its overall performance (and there is that brief window in step a where we take a read-only table level lock to get a GTIDset/position to go along with the consistent snapshot of the table). It's for this reason that we recommend you setup a standard MySQL replica — if you don't already have one — and use that as the source MySQL instance for the migration. This is another key factor that ensures we not only avoid downtime, butwe avoid any impact whatsoever on the live production system that is currently serving your application data. Once the data has been copied to the new system, we continue to replicate changes from the old database instance to the PlanetScale database so that we are ready for the cutover at any point. Again, there is a stream from the source (MySQL and unmanaged tablet pair) to each target shard's PRIMARY tablet. For each stream, the tablet on the source will connect to the external MySQL instance and initiate aCOM_BINLOG_DUMP_GTID protocol command, providing the GTID executed snapshot that the migration/workflow has which corresponds to where we were when the copy phase completed (seeLife of a Stream for more details). Each replication stream then continues to filter those binlog events based on the sharding scheme and forward them to the target shard's PRIMARY tabletwhere the changes are applied and persistent metadata is stored (in the sidecar database's vreplication table) to record the associated GTID set/position so thatno matter what happens, we can restart our work and pick up where we left off. We continue to do this until we're ready to cutover. Run a VDiff to verify that all of the data has been correctly copied and that the new system is in sync with the old, identifying any discrepancies that need to be addressed before the cutover (see Introducing VDiff V2for more details) — also done without incurring any downtime. Each table in the workflow is diffed serially before the VDiff is complete. So all steps below are done for each table in the workflow. a. We first get a named lock on the workflow in the target keyspace — in the topology server — to prevent any concurrent changes to the workflow while we are initializing the VDiff as we will be manipulating the workflow to stop it, update it, and then restart it.Once we have the named lock on the workflow we stop the workflow for the table diff initialization done in steps a through e. b. We then connect to the source (MySQL and unmanaged tablet pair) and initiate a consistent snapshot there to use for the comparision in the same way that we did in step 1a for the data copy. At this point we have the snapshotwe need on the source side. c. We then use the GTID position/snapshot from step b to start the stream on each target PRIMARY tablet until it has reached that given position and then it stops (this is the same thing a standard MySQL instance does for START REPLICA UNTIL). On eachtarget shard we then setup a consistent snapshot just as we did on the source side. Now we have a consistent snapshot of the table on the source instance and each target shard that we can use for the data comparison. d. Now we restart the VReplication workflow so that it can continue replicating changes from the source instance to the target shards. e. At this point, we are done manipulating the workflow for the table diff, and we release the named lock on the workflow in the target keyspace taken in step a. f. We then execute a full table scan on the source instance and all target shards, comparing the streamed results as we go along, noting any discrepancies as they are encountered (a row missing on either side or a row with different values) — the state of the diff being persisted in the sidecar database's VDiff tables on each target shard. You can follow the progress as it goes, which includes an ETA, and when it's done you can see a detailed report which notes if any discrepancies were found and providing details on what those differences were — allowing you to address them before the cutover (see theVDiff show command). The VDiff will choose REPLICA tablets by default on the source and target, for the data streaming (the work is still orchestrated by and the state still stored on the target PRIMARY tablets), to prevent any impact on the live production system. The VDiff is also fault-tolerant — it will automatically pickup where it left off if any error is encountered — and it can be done in an incremental fashion so that if e.g. you are in the pre-cutover state for many weeks or even months, you can run an initial VDiff, andthen resume that one as you get closer to the cutover point. While it is not required that this step is taken, it is highly recommended that at least one VDiff is run before the cutover to ensure that the data has been copied correctly and that the new system is in sync with the old. At some point between steps 1 and 4, the application starts sending traffic to PlanetScale rather than directly to their old system. PlanetScale then continues to route traffic back to the old system until we're ready to cutover. Schema routing rules are put in place so that during the migration, queries against the tables being migrated will be routed to the correct destination — the external MySQL instance (old system) or the PlanetScale database (new system) depending on where we are in the migration process. When the migration starts, these rules ensure that all queries are sent to the source keyspace (old system) and they are updated accordingly when traffic is cutover along with if and when the cutover is reversed. You can remain in this state as long as necessary as you prepare for the application cutover and perform additional testing of the application and database system. Transparently cutover application traffic to the new system as application traffic is going through PlanetScale and PlanetScale will now route traffic to the new internal system rather than the old external one. a. Under the hood, the MoveTables SwitchTraffic command is executed for the migration workflow. b. It will first do some pre-checks to ensure that the traffic switch should succeed, such as checking the overall health of the tablets involved, the replication lag for the workflow (as the workflow has to fully catch up with the source before we can do the traffic switch and there's a timeout for that since this should be abrief period of time as the queries are being buffered), and other necessary state across the cluster. If everything looks good then we will proceed with the actual traffic switch. c. We ensure that there are viable PRIMARY tablets in the source keyspace necessary to setup the reverse VReplication workflow which we will put in place when the traffic switch is complete so that the old system continues to stay in sync with the new and we can cut the traffic backover to the old system if needed for any reason. This offers even more flexibility and confidence as if any unexpected errors or performance issues occur (keep in mind that you may be going from one MySQL, or even MariaDB, version to another and from an unsharded database to a sharded one) then you can quicklyrevert the cutover and investigate the issue. Then once the issues are addressed you can attempt to cut the traffic back over to the new system again — without the pressure of the system being down or needing to complete the final cutover by anyparticular time. d. We take a lock on the source and target keyspace in the topology server to prevent concurrent changes to these keyspaces in the cluster, along with a named lock on the workflow in the target keyspace to prevent concurrent changesto the workflow itself. e. We stop writes on the source keyspace and begin buffering the incoming queries (see VTGate Buffering for more details) so that they can be executed on the target keyspace once the traffic switch is complete. f. We wait for replication in the workflow to fully catch up so that the target keyspace has every write performed against the source and nothing is lost. g. We create a reverse VReplication workflow that will replicate changes from the target keyspace (new system) back to the source keyspace (old system). This is the workflow that ensures that the old system is kept in sync with writes to the new system in case we need to revert the cutover for any reason (using theMoveTables ReverseTraffic command). h. We initialize any Vitess Sequences that are being used in the target keyspace. This is done to seamlessly replace auto_increment usage, when the tables are being sharded as part of the migration, to provide the same functionality of auto generating incrementing unique values in a sharded environment. i. We allow writes to the target keyspace. j. We update the schema routing rules so that any queries against the tables being migrated will now be routed to the target keyspace (new system). k. We start the reverse VReplication workflow created in step g. l. We mark the original VReplication workflow as Frozen so that it is hidden and cannot be manipulated but we retain that information and state. m. We release the keyspace and named locks taken in step a. You can remain in this state for as long as you like. It's only when you are 100% confident in the migration and no longer need the option of cutting traffic back over to the old system that you can proceed to complete the migration, which uses the MoveTables complete command,to clean up the workflow and all of its migration related artifacts that were put in place (such as the routing rules). See How Traffic Is Switched for additional details. All of this work is done in a fault-tolerant way. This means that anything can fail throughout this process and the system will be able to recover and continue where it left off. This is critical for data imports at a certain scale where things can take many hours, days, or even weeks to complete andthe likelihood of encountering some type of error — even an ephemeral network or connection related error across the fleet of processes involved in the migration — becomes increasingly likely. Conclusion Data migrations are a critical part of the lifecycle of any database system. They are sometimes necessary for upgrading to new versions of your existing database system, sharding your existing database system, or moving to an entirely new database system. You've likely been involved in past migrations thathave caused downtime or other issues and may be thinking about the next migration you need to do and how you can avoid those issues. In walking through how we perform data migrations at PlanetScale we hope that you can see ways to improve your own data migrations and avoid various pitfalls and issues that can lead to undesirable outcomes. We're happy to help you with your next data migration — directly as a customer through our work, or indirectly as a member of ourshared database community through the sharing of information and practices as we've done here. Happy migrations!]]> Faster backups with sharding https://planetscale.com/blog/faster-backups-with-sharding 2024-07-30T00:00:00.000Z 2024-07-30T00:00:00.000Z Ben Dicken Building data pipelines with Vitess https://planetscale.com/blog/building-data-pipelines-with-vitess 2024-07-29T15:00:00.000Z 2024-07-29T15:00:00.000Z Matt Lord The State of Online Schema Migrations in MySQL https://planetscale.com/blog/state-of-online-schema-migrations-in-mysql 2024-07-23T00:00:00.000Z 2024-07-23T00:00:00.000Z Shlomi Noach Optimizing aggregation in the Vitess query planner https://planetscale.com/blog/optimizing-aggregation-in-the-vitess-query-planner 2024-07-22T00:00:00.000Z 2024-07-22T00:00:00.000Z Andres Taylor Dealing with large tables https://planetscale.com/blog/dealing-with-large-tables 2024-07-10T00:00:00.000Z 2024-07-10T00:00:00.000Z Ben Dicken Sharding strategies: directory-based, range-based, and hash-based https://planetscale.com/blog/types-of-sharding 2024-07-08T15:00:00.000Z 2024-07-08T15:00:00.000Z Holly Guevara Announcing Vitess 20 https://planetscale.com/blog/announcing-vitess-20 2024-06-27T09:01:00.000Z 2024-06-27T09:01:00.000Z Vitess Engineering Team 5 LIMIT 1; Multi-table updates and multi-target updates enhance flexibility:UPDATE t1 JOIN t2 ON t1.id = t2.id JOIN t3 ON t1.col = t3.col SET t1.baz = 'abc', t1.apa = 23 WHERE t3.foo = 5 AND t2.bar = 7; UPDATE t1 JOIN t2 ON t1.id = t2.id SET t1.foo = 'abc', t2.bar = 23; Advanced delete operations with subqueries and multi-target support are included:DELETE FROM t1 WHERE id IN (SELECT col FROM t2 WHERE foo = 32 AND bar = 43); DELETE t1, t3 FROM t1 JOIN t2 ON t1.id = t2.id JOIN t3 ON t1.col = t3.col; These features provide greater control and efficiency for managing sharded data. For more details, please refer to the Vitess and MySQL documentation. VReplication: multi-tenant imports (experimental) Many web-scale applications use a multi-tenant architecture where each tenant has their own database (with identical schemas). There are several challenges with this approach — like provisioning and scaling potentially tens of thousands of databases and uniformly updating database schemas across them. A sharded Vitess keyspace is a great option for such a system with a single logical database serving all tenants. Vitess 20 adds support for importing data from such a multi-tenant setup into a single Vitess keyspace, with new --shards and --tenant-id flags for the MoveTables workflow. You would run one such workflow for each tenant, with imported tenants being served by the Vitess cluster. Online DDL improvements Vitess migrations now support enum definition reordering. Vitess opts to use enums by alias (their string representation) rather than by ordinal value (the internal integer representation). Vitess now has better analysis for INSTANT DDL scenarios, enabled with the --prefer-instant-ddl DDL strategy flag. It is able to predict whether a migration can be fulfilled by the INSTANT algorithm and use this algorithm if so. It also improves support for range partitioning migrations, and opts to use direct partitioning queries over Online DDL where appropriate. VDiffs can now be run on Online DDL workflows that are still in progress (i.e., not yet cut-over). Release 20.0 drops support for gh-ost for Online DDL, as we continue to invest in vitess migrations based on VReplication. The gh-ost strategy is still recognized; however: Vttablet binaries no longer bundle the gh-ost binary. The user should provide their own gh-ost binary, and supply vttablet --gh-ost-path. Vitess no longer tests gh-ost in CI/end-to-end tests. Vitess-operator Automated and scheduled backups are now available as an experimental feature in v2.13.0. We have added a new user guide for this feature. Vitess and the community As an open-source project, Vitess thrives on the contributions, insights, and feedback from the community. Your experiences and input are invaluable in shaping the future of Vitess. We encourage you to share your stories and ask questions on GitHub or in the Slack Vitess community. Getting started For a seamless transition to Vitess 20, we highly recommend reviewing the detailed release notes. Additionally, you can explore the Vitess documentation for guides, best practices, and tips to make the most of Vitess 20. Whether you're upgrading from a previous version or running Vitess for the first time, our resources are designed to support you every step of the way. Thank you for your support and contributions to the Vitess project!]]> Self-managed Vitess vs Managed Vitess with PlanetScale https://planetscale.com/blog/self-run-vs-managed-vitess-with-planetscale 2024-05-24T14:00:00.000Z 2024-05-24T14:00:00.000Z Holly Guevara Achieving data consistency with the consistent lookup Vindex https://planetscale.com/blog/vitess-consistent-lookup-vindex 2024-04-29T00:00:00.000Z 2024-04-29T00:00:00.000Z Harshit Gangal Deepthi Sigireddi The MySQL adaptive hash index https://planetscale.com/blog/the-mysql-adaptive-hash-index 2024-04-24T00:00:00.000Z 2024-04-24T00:00:00.000Z Ben Dicken CREATE INDEX alias_index ON user(username) USING HASH; Query OK, 0 rows affected, 1 warning (0.79 sec) Records: 0 Duplicates: 0 Warnings: 1 However, the information_schema tells us that it really is a B-tree index that was created:SELECT table_name, index_name, index_type FROM information_schema.statistics WHERE table_schema = 'quiz' AND table_name = 'user'; +------------+-------------+------------+ | TABLE_NAME | INDEX_NAME | INDEX_TYPE | +------------+-------------+------------+ | user | alias_index | BTREE | | user | PRIMARY | BTREE | +------------+-------------+------------+ 2 rows in set (0.00 sec) It is a bit unfortunate that InnoDB does not support building on-disk, hash based indexes, as this type could be useful and more performant than a B-tree in some instances.However, this lack of support does not mean that hashing is not used at all for index lookups. The adaptive hash index Though InnoDB does not support on-disk hash indexes, MySQL has a feature to do in-memory hash lookups for indexing.This is known as the adaptive hash index.This feature can be used to speed up already-fast B-tree lookups, accelerating the performance of the queries that utilize these indexes as a part of their query plan.This acts as a layer that sits between the execution of MySQL and the in-memory buffer pool. As its name suggests, the adaptive hash index (AHI) is constructed at runtime, and its usage adapts to the characteristics of your workload.If MySQL observes that a particular value is getting repeatedly looked up in a B-tree index, an entry in the AHI can be created either with the full value, or a prefix of the value.For future lookups of this same value (which are likely, since MySQL observed its repeated use) it will use the AHI instead of the B-tree.The keys of the AHI are the values (or value prefixes) of the underlying index.The values are pointers, referring to where the data for this value lives within the InnoDB buffer pool. The pointers in the adaptive hash index only point to data within the buffer pool.Thus, the buffer pool needs to be sufficiently large for the AHI to kick in.If it is small and there are a lot of evictions taking place, it is not worth using it.Thankfully, MySQL is able to automatically adjust its use of the AHI based on the behavior it observes in the buffer pool.If conditions are not right for its use (few repeated lookups, small buffer pool, etc), MySQL will reduce or eliminate its use. Though it can speed up queries, there is a bit of overhead to maintaining this special hash index.The feature can be enabled or disabled via the innodb_adaptive_hash_index configuration option.It is typically enabled by default, but if you have a workload that you know will not benefit from it, it can be disabled using innodb_adaptive_hash_index=0 in your configuration file. Testing performance of the adaptive hash index Let's run a few tests to see how the adaptive hash index can help a workload.We'll start with something very simple.I'll execute the following query 500k times in two different scenarios: with and without the AHI enabled.SELECT user_id, username, bio FROM user WHERE username = 'willpeace1'; This is executing against a users table with a little over 390 million rows in it.With it disabled, we get the following timing:$ python3 same_query.py 500000 1 starting query load completed in 35.6 seconds QPS = 14043.57 This goes fast, because I have a B-tree index already set on the username column.While this workload was running, I executed SHOW ENGINE INNODB STATUS \G;.This command provides information about what is going on with the InnoDB storage engine.You can inspect the INSERT BUFFER AND ADAPTIVE HASH INDEX section to see if and how much the adaptive hash index is being used.------------------------------------- INSERT BUFFER AND ADAPTIVE HASH INDEX ------------------------------------- Ibuf: size 11826, free list len 13206, seg size 25033, 8599 merges merged operations: insert 52823, delete mark 0, delete 0 discarded operations: insert 0, delete mark 0, delete 0 Hash table size 276707, node heap has 0 buffer(s) ... 0.00 hash searches/s, 418334.67 non-hash searches/s The key line to observe here is the last one, which indicates that no hash lookups are occurring.This is expected, since we have disabled the feature. Now after enabling the adaptive hash index and restarting the server, let's try again:$ python3 same_query.py 500000 1 starting query load completed in 29.94 seconds QPS = 16701.1 We achieved about a 16% speed up.This is a small, but still quite useful performance boost.B-tree index lookups are already very fast, but we have layered an additional optimization on top to make it even faster.I grabbed information about the buffer pool while this ran, and this shows that the AHI was being used:------------------------------------- INSERT BUFFER AND ADAPTIVE HASH INDEX ------------------------------------- Ibuf: size 11445, free list len 13587, seg size 25033, 1881 merges merged operations: insert 11507, delete mark 0, delete 0 discarded operations: insert 0, delete mark 0, delete 0 Hash table size 276707, node heap has 3 buffer(s) ... 350953.05 hash searches/s, 50985.01 non-hash searches/s Now, many hash searches are occurring.There are still some non-hash searches happening as well, which makes sense since the query still needs to access the actual data in the row, not just the index value. Let's try another workload.This time, we'll re-use the same query, except instead of always searching for the same username, on each execution we'll randomly select one username from a pool of one thousand.Though this specific workload is unrealistic, this can be thought of as a database and workload with a large amount of cold data, and a small amount of hot data. Without the adaptive hash index disabled, we get:$ python3 load.py 500000 1 --- starting query workload completed in 54.16 seconds QPS = 9231.62 With it enabled, we instead see:$ python3 load.py 500000 1 --- starting query workload completed in 43.24 seconds QPS = 11562.05 In this workload, we got a 20% performance improvement. These tests were executed on on a table with over 390 million rows:mysql> SELECT count(*) FROM user; +-----------+ | count(*) | +-----------+ | 398748007 | +-----------+ Even with such a large data set, the B-tree index for the username column was only 4 levels deep (this was checked with the help of innodb_ruby).Running these types of workloads on tables with smaller indexes, and therefore shorter B-tree indexes, may result in less noticeable speedup.On the other hand, workloads using deeper B-tree indexes may see even more performance improvement. Conclusion When dealing with large, multi-terabytes databases and workloads with hundreds of thousands of queries per second, even small improvements like this can have a big impact on query latency and database server infrastructure needs.The adaptive hash index provides such improvements for already-fast B-tree index lookups, helping certain types of workloads get a boost in performance.Whether or not the AHI will help your workload depends heavily on data access patterns and the size of your InnoDB buffer pool.]]> Introducing global replica credentials https://planetscale.com/blog/introducing-global-replica-credentials 2024-04-17T16:01:00.000Z 2024-04-17T16:01:00.000Z Matt Robenolt Iheanyi Ekechukwu cluster, the Credential contains the rest of the information needed to fully connect to the underlying database with the correct ACLs and to which TabletType. The Endpoint Now the endpoint is where the first bit of magic happens. As you may have noticed in the product, we surface two different hostname options, a "Direct" and an "Optimized". The "Direct" has the form of {region}.connect.psdb.cloud and the "Optimized" is of the form {provider}.connect.psdb.cloud. The Direct endpoint is the most straightforward, and represents the Edge node in that region explicitly. You can choose really any region you'd like as the first hop to route through and you'll still get to the correct destination, but we give you the endpoint that is closest to the database, not the endpoint closest to you. But really, you can pick any public endpoint in the same provider if you're clever. The Optimized endpoint is backed by a latency-based DNS resolver. In AWS, for example, this is their Route53 latency-based routing policy. Which is most of the magic to resolve aws.connect.psdb.cloud to the nearest edge region to you. This means whether you're connecting from your local machine with pscale connect or from the datacenter next to your database, you get routed through the closest region to you, which gives us the CDN effect. Putting this together With these three bits, we can put together the story for how we can route to any of your replicas geographically. Starting with the initial connection pool at our edge, this applies exactly the same to a connection over HTTP and MySQL protocol. Once a connection is established to us, regardless of where your database is located, your connection is terminated at this same edge node in our network. When a new region is added, the underlying Route is mutated to add the new cluster. Since we maintain warm connections between all of our regions ready to go, we utilize these to measure latency continuously as a part of regular health checking. So, for example, the us-east-1 edge node is continuously pinging its peers, similar to a mesh network and measuring their latency. Once a Route is seen over the etcd watcher, before it's accessible to being used, we are able to simply sort the list of clusters based on their latency times we already are tracking. We periodically re-sort every Route if/when latency values change. This keeps the "next hop" decision always clusters[0] in practice. In the event if a hard failure (if for some reason this entire region were down), we could go over to the next option if there were multiple choices. Ultimately, because the connection is already established with us during all of this, the Route is utilized on a per-query basis, thus without needing to reconnect or anything, we can route you to the lowest latency next hop in realtime. Similarly, when read-only regions are added and removed, we only need to mutate this Route with a new set of what regions your database is in, and we just maintain a sorted list ready to go.]]> Profiling memory usage in MySQL https://planetscale.com/blog/profiling-memory-usage-in-mysql 2024-04-11T00:00:00.000Z 2024-04-11T00:00:00.000Z Ben Dicken | +---------------------------------------+---------------+ However, the name indicates that it is memory being used for sorting data from a file.This makes sense as a large part of the expense of this query would be sorting the data so that is can be displayed in descending order. Collecting usage over time As a next step, we need to be able to sample this memory usage over time.For short queries this will not be as useful, as we'll only be able to execute this query once, or a small number of times while the profiled query is executing.This will be more useful for longer-running queries, ones that take multiple seconds or minutes.These, would be the types of queries we'd want to profile anyways, as these are the ones likely to use a large portion of memory. This could be implemented fully in SQL and invoked via a stored procedure.However, in this case, let's use a separate script in Python to provide monitoring.#!/usr/bin/env python3 import time import MySQLdb import argparse MEM_QUERY=''' SELECT event_name, current_number_of_bytes_used FROM performance_schema.memory_summary_by_thread_by_event_name WHERE thread_id = %s ORDER BY current_number_of_bytes_used DESC LIMIT 4 ''' parser = argparse.ArgumentParser() parser.add_argument('--thread-id', type=int, required=True) args = parser.parse_args() dbc = MySQLdb.connect(host='127.0.0.1', user='root', password='password') c = dbc.cursor() ms = 0 while(True): c.execute(MEM_QUERY, (args.thread_id,)) results = c.fetchall() print(f'\n## Memory usage at time {ms} ##') for r in results: print(f'{r[0][7:]} -> {round(r[1]/1024,2)}Kb') ms+=250 time.sleep(0.25) This is a simple, first stab at such a monitoring script.In summary, this code does the following: Get the provided thread ID to monitor via command line Set up a connection to a MySQL database Every 250 milliseconds, execute a query to get the top 4 used memory categories and print a readout This could be adjusted in many ways depending on your profiling needs.For example, tweaking the frequency of the ping to the server or changing how many memory categories are listed per iteration.Running this while a query is executing provides results like this:... ## Memory usage at time 4250 ## innodb/row0sel -> 25.22Kb sql/String::value -> 16.07Kb sql/user_var_entry -> 0.41Kb innodb/memory -> 0.23Kb ## Memory usage at time 4500 ## innodb/row0sel -> 25.22Kb sql/String::value -> 16.07Kb sql/user_var_entry -> 0.41Kb innodb/memory -> 0.23Kb ## Memory usage at time 4750 ## innodb/row0sel -> 25.22Kb sql/String::value -> 16.07Kb sql/user_var_entry -> 0.41Kb innodb/memory -> 0.23Kb ## Memory usage at time 5000 ## innodb/row0sel -> 25.22Kb sql/String::value -> 16.07Kb sql/user_var_entry -> 0.41Kb innodb/memory -> 0.23Kb ... This is great, but there's a few weaknesses.It would be nice to see more than the top 4 memory usage categories, but increasing that numbers increases the size of this already-large output dump.It would also be nice to have an easier way to get a picture of the memory usage at-a-glance via some visualizations.This could be done by having the script dump the results to a CSV or JSON, and then loading them up later in a visualization tool.Even better, we could plot the results we are getting live, as the data is streaming in.This provides a more up-to-date view, and allows us to observe the memory usage live as it is happening, all in one tool. Plotting memory usage In order make this tool even more useful and provide visualizations, a few changes are going to be made. The user will provide connection ID on the command line, and the script will be responsible for finding the underlying thread. The frequency at which the script requests memory data will be configurable, also via the command line. The matplotlib library will be used to generate a visualization of the memory usage.This will consist of a stack plot with a legend showing the top memory usage categories, and will retain the past 50 samples. It's quite a bit of code, but is included here for the sake of completeness.#!/usr/bin/env python3 import matplotlib.pyplot as plt import numpy as np import MySQLdb import argparse MEM_QUERY=''' SELECT event_name, current_number_of_bytes_used FROM performance_schema.memory_summary_by_thread_by_event_name WHERE thread_id = %s ORDER BY event_name DESC''' TID_QUERY=''' SELECT thread_id FROM performance_schema.threads WHERE PROCESSLIST_ID=%s''' class MemoryProfiler: def __init__(self): self.x = [] self.y = [] self.mem_labels = ['XXXXXXXXXXXXXXXXXXXXXXX'] self.ms = 0 self.color_sequence = ['#ffc59b', '#d4c9fe', '#a9dffe', '#a9ecb8', '#fff1a8', '#fbbfc7', '#fd812d', '#a18bf5', '#47b7f8', '#40d763', '#f2b600', '#ff7082'] plt.rcParams['axes.xmargin'] = 0 plt.rcParams['axes.ymargin'] = 0 plt.rcParams["font.family"] = "inter" def update_xy_axis(self, results, frequency): self.ms += frequency self.x.append(self.ms) if (len(self.y) == 0): self.y = [[] for x in range(len(results))] for i in range(len(results)-1, -1, -1): usage = float(results[i][1]) / 1024 self.y[i].append(usage) if (len(self.x) > 50): self.x.pop(0) for i in range(len(self.y)): self.y[i].pop(0) def update_labels(self, results): total_mem = sum(map(lambda e: e[1], results)) self.mem_labels.clear() for i in range(len(results)-1, -1, -1): usage = float(results[i][1]) / 1024 mem_type = results[i][0] # Remove 'memory/' from beginning of name for brevity mem_type = mem_type[7:] # Only show top memory users in legend if (usage < total_mem / 1024 / 50): mem_type = '_' + mem_type self.mem_labels.insert(0, mem_type) def draw_plot(self, plt): plt.clf() plt.stackplot(self.x, self.y, colors = self.color_sequence) plt.legend(labels=self.mem_labels, bbox_to_anchor=(1.04, 1), loc="upper left", borderaxespad=0) plt.xlabel("milliseconds since monitor began") plt.ylabel("Kilobytes of memory") def configure_plot(self, plt): plt.ion() fig = plt.figure(figsize=(12,5)) plt.stackplot(self.x, self.y, colors=self.color_sequence) plt.legend(labels=self.mem_labels, bbox_to_anchor=(1.04, 1), loc="upper left", borderaxespad=0) plt.tight_layout(pad=4) return fig def start_visualization(self, database_connection, connection_id, frequency): c = database_connection.cursor(); fig = self.configure_plot(plt) while(True): c.execute(MEM_QUERY, (connection_id,)) results = c.fetchall() self.update_xy_axis(results, frequency) self.update_labels(results) self.draw_plot(plt) fig.canvas.draw_idle() fig.canvas.start_event_loop(frequency / 1000) def get_command_line_args(): ''' Process arguments and return argparse object to caller. ''' parser = argparse.ArgumentParser(description='Monitor MySQL query memory for a particular connection.') parser.add_argument('--connection-id', type=int, required=True, help='The MySQL connection to monitor memory usage of') parser.add_argument('--frequency', type=float, default=500, help='The frequency at which to ping for memory usage update in milliseconds') return parser.parse_args() def get_thread_for_connection_id(database_connection, cid): ''' Get a thread ID corresponding to the connection ID PARAMS database_connection - Database connection object cid - The connection ID to find the thread for ''' c = database_connection.cursor() c.execute(TID_QUERY, (cid,)) result = c.fetchone() return int(result[0]) def main(): args = get_command_line_args() database_connection = MySQLdb.connect(host='127.0.0.1', user='root', password='password') connection_id = get_thread_for_connection_id(database_connection, args.connection_id) m = MemoryProfiler() m.start_visualization(database_connection, connection_id, args.frequency) connection.close() if __name__ == "__main__": main() With this, we can do detailed monitoring of executing MySQL queries.To use it, first get the connection ID for the connection you want to profile:SELECT CONNECTION_ID(); Then, executing the following will begin a monitoring session:./monitor.py --connection-id YOUR_CONNECTION_ID --frequency 250 When executing a query on the database, we can observe the increase in memory usage, and see what categories of memory are the largest contributors. This visualization can also help us to clearly see what kinds of operations are memory hogs.For example, here is a snippet of a memory profile for creating a FULLTEXT index on a large table: The memory usage is significant, and continues to grow into using hundreds of megabytes as it executes. For another example of how you can use MySQL to profile memory usage, see check out this DBAMA presentation and the corresponding GitHub repository. Conclusion Though it may not be needed as often, having the ability to get detailed memory usage information can be extremely valuable when the need for detailed query optimization arises.Doing this can reveal when and why MySQL may be cause memory pressure on the system, or if a memory upgrade for your database server may be needed.MySQL provides a number of primitives that you can build upon to develop profiling tooling for your queries and workload.]]> Summer 2023: Fuzzing Vitess at PlanetScale https://planetscale.com/blog/summer-2023-fuzzing-vitess-at-planetscale 2024-04-09T00:00:00.000Z 2024-04-09T00:00:00.000Z Arvind Murty How PlanetScale makes schema changes https://planetscale.com/blog/how-planetscale-makes-schema-changes 2024-04-04T09:00:00.000Z 2024-04-04T09:00:00.000Z Mike Coutermarsh Identifying and profiling problematic MySQL queries https://planetscale.com/blog/identifying-and-profiling-problematic-mysql-queries 2024-03-29T00:00:00.000Z 2024-03-29T00:00:00.000Z Ben Dicken | 2574002473 | +---------------+-------------+------------+------------+ This indicated that there have been over 2 billion row reads that were not fulfilled by an index (the row where INDEX_NAME is ).Either this table needs one or more indexes added to it, or the queries using this table need to be updated, or both! There's also a bunch of cool stats you can look at over in the sys table.Here's one quick example.You can grab data on how many full table scans are being executed by each query using the sys.statements_with_full_table_scans table.USE sys; SELECT query, db, exec_count, total_latency FROM sys.statements_with_full_table_scans ORDER BY exec_count DESC LIMIT 5; +-------------------------------------------------------------------+------+------------+---------------+ | query | db | exec_count | total_latency | +-------------------------------------------------------------------+------+------------+---------------+ | SELECT `class` , `size` FROM `spaceship` WHERE `size` > ? | game | 8422 | 26.65 s | | SELECT `p1` . `username` , `m` ... ` WHERE `m` . `created_at` > ? | game | 6742 | 6.45 min | | SELECT * FROM `earned_achievem ... id` AND `ea` . `player_id` > ? | game | 6718 | 969.17 ms | | SELECT * FROM `item` WHERE NAME LIKE ? LIMIT ? | game | 5676 | 969.43 ms | | SELECT NAME , `size` FROM `planet` WHERE `population` > ? | game | 5625 | 173.02 ms | +-------------------------------------------------------------------+------+------------+---------------+ This shows which queries are triggering full table scans.In this case, there's a ton, but this would be expected since there are no indexes on this example database other than the default ones on the primary keys. For more information about how to work with the performance_schema and sys tables, check out this video. Inspecting with EXPLAIN By this point, you've hopefully gathered a collection of queries that need to be further inspected.The next step is to do some root-cause analysis into why these queries are taking a long time and reading too many of rows.Of course, one great way to drill into the behavior of a query is with EXPLAIN or EXPLAIN ANALYZE.In this example, I'm going to use EXPLAIN ANALYZE. For example, running the following query:EXPLAIN ANALYZE SELECT p1.username, m.to_id, p2.username, m.from_id FROM message m LEFT JOIN player p1 ON m.to_id = p1.id LEFT JOIN player p2 ON m.from_id = p2.id WHERE m.created_at > '2020-10-10 00:00:00'; Gives detailed information regarding the query plan and costs of the various steps.+--------------------------------------------------------------------------------------------------------------------------------------------+ | EXPLAIN | +--------------------------------------------------------------------------------------------------------------------------------------------+ | -> Nested loop left join (cost=333272 rows=332036) (actual time=0.515..320 rows=345454 loops=1) | | -> Nested loop left join (cost=217059 rows=332036) (actual time=0.475..240 rows=345454 loops=1) | | -> Filter: (m.created_at > TIMESTAMP'2020-10-10 00:00:00') (cost=100846 rows=332036) (actual time=0.153..188 rows=345454 loops=1) | | -> Table scan on m (cost=100846 rows=996208) (actual time=0.148..159 rows=1e+6 loops=1) | | -> Single-row index lookup on p1 using PRIMARY (id=m.to_id) (cost=0.25 rows=1) (actual time=54.5e-6..70.9e-6 rows=1 loops=345454) | | -> Single-row index lookup on p2 using PRIMARY (id=m.from_id) (cost=0.25 rows=1) (actual time=137e-6..153e-6 rows=1 loops=345454) | +--------------------------------------------------------------------------------------------------------------------------------------------+ There are a number of things to look at when inspecting this output.Generally, if you see table scans over large tables, this is something you should try to mitigate with an index.More broadly, any time you see a node with a high cost or rows value, this probably deserves further tuning.Perhaps the query can be re-written, or perhaps one or more indexes can help improve performance. For more information about EXPLAIN, see our MySQL for Developers course or our How to read MySQL EXPLAINS blog post. Preparing to instrument a query Explain is a great tool, but you can also profile a query to determine how much time it spends in each stage of execution.To do this, first ensure that the proper instruments and consumers are enabled so that information can be gathered appropriately.In order to do this, run the following:UPDATE performance_schema.setup_instruments SET ENABLED = 'YES', TIMED = 'YES'; UPDATE performance_schema.setup_consumers SET ENABLED = 'YES', TIMES = 'YES'; If you'd like, you can be more selective at this step.Rather than enabling this setting for all of instruments and consumers, you can enable subsets of the rows in these two tables.To enable it for the stages of query execution, you could run the following.UPDATE performance_schema.setup_instruments SET ENABLED = 'NO', TIMED = 'NO'; UPDATE performance_schema.setup_instruments SET ENABLED = 'YES', TIMED = 'YES' WHERE NAME LIKE '%stage/%'; Ultimately, it's up to you based on what level of comfort you have with possible performance hits of profiling.If you're not very concerned with the profiling overhead, just turn everything on and then disable later when you're finished. Next, ensure that history tracking is enabled.If you have not configured it before, you'll probably see the following when you look at the setup_actors configuration:SELECT * FROM performance_schema.setup_actors; +------+------+------+---------+---------+ | HOST | USER | ROLE | ENABLED | HISTORY | +------+------+------+---------+---------+ | % | % | % | YES | YES | +------+------+------+---------+---------+ If you want you can leave these settings as-is.However, this means that performance schema and history tracking will be enabled for all users.This could have a (small) adverse effect on the overall performance of your system.You can optionally enable it only for one specific user (this would be the user you need to run your test queries on).If you choose to go this route, turn it off globally:UPDATE performance_schema.setup_actors SET ENABLED = 'NO', HISTORY = 'NO' WHERE HOST = '%' AND USER = '%'; And then enable it only for the user(s) that you want to track for:INSERT INTO performance_schema.setup_actors (HOST, USER, ROLE, ENABLED, HISTORY) VALUES ('your_host', 'your_user', '%', 'YES', 'YES'); Profiling the query Now, let's profile one of our problematic query.First, get the ID of the connection you are going to run the query on.SET @connection_thread = ( SELECT thread_id FROM performance_schema.threads WHERE PROCESSLIST_ID = CONNECTION_ID() ); Next, execute the query you want to profile.Immediately after doing this, run the following query to determine the starting event ID of that execution:SELECT thread_id, statement_id, SUBSTRING(sql_text,1,50) FROM performance_schema.events_statements_history_long WHERE thread_id = @connection_thread ORDER BY event_id DESC LIMIT 20; Find the statement you want to profile and put its ID into a variable.SET @statement_id = ?; Finally, run the following to see the profiling information for that query execution.SET @eid = (SELECT event_id FROM performance_schema.events_statements_history_long WHERE statement_id = @statement_id); SET @eeid = (SELECT end_event_id FROM performance_schema.events_statements_history_long WHERE statement_id = @statement_id); SELECT event_name, source, (timer_end-timer_start)/1000000000 as 'milliseconds' FROM performance_schema.events_stages_history_long WHERE event_id BETWEEN @eid AND @eeid; This should provide a timing breakdown that looks something like this:+------------------------------------------------+----------------------------------+--------------+ | event_name | source | milliseconds | +------------------------------------------------+----------------------------------+--------------+ | stage/sql/starting | init_net_server_extension.cc:110 | 0.2400 | | stage/sql/Executing hook on transaction begin. | rpl_handler.cc:1481 | 0.0010 | | stage/sql/starting | rpl_handler.cc:1483 | 0.0110 | | stage/sql/checking permissions | sql_authorization.cc:2169 | 0.0020 | | stage/sql/checking permissions | sql_authorization.cc:2169 | 0.0000 | | stage/sql/checking permissions | sql_authorization.cc:2169 | 0.0030 | | stage/sql/Opening tables | sql_base.cc:5859 | 0.0950 | | stage/sql/init | sql_select.cc:759 | 0.0050 | | stage/sql/System lock | lock.cc:331 | 0.0110 | | stage/sql/optimizing | sql_optimizer.cc:355 | 0.0270 | | stage/sql/statistics | sql_optimizer.cc:699 | 0.1040 | | stage/sql/preparing | sql_optimizer.cc:783 | 0.0590 | | stage/sql/executing | sql_union.cc:1676 | 735.3020 | | stage/sql/end | sql_select.cc:795 | 0.0010 | | stage/sql/query end | sql_parse.cc:4805 | 0.0020 | | stage/sql/waiting for handler commit | handler.cc:1636 | 0.0060 | | stage/sql/closing tables | sql_parse.cc:4869 | 0.0080 | | stage/sql/freeing items | sql_parse.cc:5343 | 0.2810 | | stage/sql/cleaning up | sql_parse.cc:2387 | 0.0010 | +------------------------------------------------+----------------------------------+--------------+ In this case, you can see that the query spent the majority of the time in the execution stage.However, this would also reveal if the query had spent a lot of time waiting on a lock or on optimizing, in which case you could dig further into the problem. PlanetScale Insights As we've seen, MySQL provides a lot of capability to drill into problematic queries in your workload, and what we've discussed here is really only scratching the surface.However, it should also be clear that gleaning this information can be tedious.Getting exactly what you want requires significant poking around and digging through tables in performance_schema and sys. Many of these same observations can be gathered much easier using PlanetScale Insights.Insights provides a plethora of useful visualizations to help you get an overview of the performance of your database and automatic detection of anomalous behavior. You can also use it to drill in on specific queries.For example, you can look at all queries executed over some window of time, and sort by different statistics such as rows read.This can help you quickly identify slow queries and ones where adding an index might be worthwhile. Insights allows you to gain a deep understanding of your workload.This gives you more time to focus on improving your queries, developing software, and working efficiently.]]> The Problem with Using a UUID Primary Key in MySQL https://planetscale.com/blog/the-problem-with-using-a-uuid-primary-key-in-mysql 2024-03-19T09:00:00.000Z 2024-03-19T09:00:00.000Z Brian Morrison II Announcing Vitess 19 https://planetscale.com/blog/announcing-vitess-19 2024-03-08T09:01:00.000Z 2024-03-08T09:01:00.000Z Vitess Engineering Team PlanetScale forever https://planetscale.com/blog/planetscale-forever 2024-03-06T09:00:00.000Z 2024-03-06T09:00:00.000Z Sam Lambert Introducing schema recommendations https://planetscale.com/blog/introducing-schema-recommendations 2024-02-28T12:00:00.000Z 2024-02-28T12:00:00.000Z Taylor Barnett Rafer Hazen Foreign key constraints are now generally available https://planetscale.com/blog/foreign-key-constraints-are-now-generally-available 2024-02-16T09:00:00.000Z 2024-02-16T09:00:00.000Z Taylor Barnett Rick Branson Amazon Aurora Pricing: The many surprising costs of running an Aurora database https://planetscale.com/blog/amazon-aurora-pricing-the-many-surprising-costs-of-running-an-aurora-database 2024-02-15T15:00:00.000Z 2026-03-09T15:00:00.000Z Brian Morrison II Three common MySQL database design mistakes https://planetscale.com/blog/three-common-mysql-database-design-mistakes 2024-02-13T09:00:00.000Z 2024-02-13T09:00:00.000Z Brian Morrison II OAuth applications are now available to everyone https://planetscale.com/blog/oauth-applications-are-now-available 2024-02-06T12:00:00.000Z 2024-02-06T12:00:00.000Z Taylor Barnett Deprecating the Scaler plan https://planetscale.com/blog/deprecating-the-scaler-plan 2024-02-05T09:00:00.000Z 2024-02-05T09:00:00.000Z Nick Van Wiggeren PlanetScale branching vs. Amazon Aurora blue/green deployments https://planetscale.com/blog/planetscale-branching-vs-amazon-aurora-blue-green-deployments 2024-02-02T09:00:00.000Z 2024-02-02T09:00:00.000Z Brian Morrison II Databases at scale https://planetscale.com/blog/databases-at-scale 2024-01-31T13:00:00.000Z 2024-01-31T13:00:00.000Z Rick Branson Considerations for building a database disaster recovery plan https://planetscale.com/blog/considerations-for-building-a-database-disaster-recovery-plan 2024-01-30T09:00:00.000Z 2024-01-30T09:00:00.000Z Brian Morrison II Working with Geospatial Features in MySQL https://planetscale.com/blog/geospatial-features-mysql 2024-01-25T09:00:00.000Z 2024-01-25T09:00:00.000Z Savannah Longoria PlanetScale vs Amazon Aurora replication https://planetscale.com/blog/planetscale-vs-aws-aurora-replication 2024-01-24T15:00:00.000Z 2024-01-24T15:00:00.000Z Brian Morrison II Introducing the Vantage and PlanetScale integration https://planetscale.com/blog/introducing-the-vantage-and-planetscale-integration 2024-01-23T12:00:00.000Z 2024-01-23T12:00:00.000Z Mike Coutermarsh MySQL isolation levels and how they work https://planetscale.com/blog/mysql-isolation-levels-and-how-they-work 2024-01-08T15:00:00.000Z 2024-01-08T15:00:00.000Z Brian Morrison II Introducing the schemadiff command line tool https://planetscale.com/blog/schemadiff-command-line-tool 2023-12-18T09:00:00.000Z 2023-12-18T09:00:00.000Z Shlomi Noach /dev/null || echo "FAIL" Showing changes The diff command can compare two schemas and write the necessary DDL to execute to get them in sync. This is how we generate our change statements when merging branches in PlanetScale.schemadiff diff --source 'myuser:mypass@tcp(127.0.0.1:3306)/test' --target /path/to/repo/source/code/schema DROP VIEW `v`; ALTER TABLE `t` MODIFY COLUMN `id` bigint; CREATE TABLE `t2` ( `id` int, `name` varchar(128) NOT NULL DEFAULT '', PRIMARY KEY (`id`) ); For more use cases, be sure to review the README for this project. The schemadiff command line tool supports MySQL 8 syntax and is released under Apache 2.0 license. We hope you find it useful!]]> $ pscale ping https://planetscale.com/blog/pscale-ping 2023-12-13T15:50:00.000Z 2023-12-13T15:50:00.000Z Matt Robenolt Announcing foreign key constraints support https://planetscale.com/blog/announcing-foreign-key-constraints-support 2023-12-05T10:00:00.000Z 2023-12-05T10:00:00.000Z Taylor Barnett The challenges of supporting foreign key constraints https://planetscale.com/blog/challenges-of-supporting-foreign-key-constraints 2023-12-05T09:00:00.000Z 2023-12-05T09:00:00.000Z Shlomi Noach Manan Gupta What is HTAP? https://planetscale.com/blog/what-is-htap 2023-12-01T17:30:00.000Z 2023-12-01T17:30:00.000Z Savannah Longoria Introducing Insights Anomalies https://planetscale.com/blog/introducing-insights-anomalies 2023-11-28T15:50:00.000Z 2023-11-28T15:50:00.000Z Rafer Hazen Webhook security: a hands-on guide https://planetscale.com/blog/securing-webhooks 2023-11-21T09:00:00.000Z 2023-11-21T09:00:00.000Z Mike Coutermarsh Three surprising benefits of sharding a MySQL database https://planetscale.com/blog/three-surprising-benefits-of-sharding-a-mysql-database 2023-11-20T15:00:00.000Z 2023-11-20T15:00:00.000Z Brian Morrison II MySQL replication: Best practices and considerations https://planetscale.com/blog/mysql-replication-best-practices-and-considerations 2023-11-15T15:00:00.000Z 2023-11-15T15:00:00.000Z Brian Morrison II A guide to HTML email with Ruby on Rails and Tailwind CSS https://planetscale.com/blog/guide-to-html-email-with-ruby-on-rails-and-tailwind-css 2023-11-14T08:01:46.798Z 2023-11-14T08:01:46.798Z Ayrton { + if (/mailer/.test(api.file.basename)) { + return { + plugins: { + 'postcss-import': {}, + 'postcss-custom-properties': { + preserve: false + }, + 'tailwindcss/nesting': {}, + tailwindcss: { + config: './tailwind.config.mailer.js' + } + } + } + } + + return { + plugins: { + 'postcss-import': {}, + 'tailwindcss/nesting': {}, + tailwindcss: {}, + autoprefixer: {} + } + } } There’s a lot to unpack here so let’s go over the above snippet step by step. First, we check if we’re post-processing the mailer CSS or application CSS. Depending on the file, we’ll post-process them slightly different to guarantee we’re optimizing for the platform. This is done in this code snippet:if (/mailer/.test(api.file.basename)) { // ... } Next, let’s look at the two import configuration rules. The first one is postcss-custom-properties:{ 'postcss-custom-properties': { preserve: false } } The preserve option determines whether Custom Properties and properties using custom properties should be preserved in their original form. We don’t want to preserve these because most email clients do not support CSS variables. We do this by setting preserve to false. For example, the two snippets below illustrate what it looks like before and after setting preserve: false: Before setting preserve: false::root { --color: red; } h1 { color: var(--color); } After setting preserve: false:h1 { color: red; } Finally, we’re telling PostCSS to use a different Tailwind config:{ tailwindcss: { config: './tailwind.config.mailer.js' } } Now that that’s all covered, let’s continue on with the Tailwind configuration. Next, create the tailwind.config.mailer.js file now:// tailwind.config.mailer.js module.exports = { content: ['app/helpers/mailer_helper.rb', 'app/views/*_mailer/*.html.erb', 'app/views/layouts/mailer.html.erb'], future: { disableColorOpacityUtilitiesByDefault: true }, theme: { extend: { borderRadius: { none: '0', xs: '2px', sm: '4px', DEFAULT: '6px', md: '8px', lg: '10px', full: '9999px' }, fontSize: { xs: '10px', sm: '12px', base: '14px', lg: '16px', xl: '18px', '2xl': '22px', '3xl': '24px', '4xl': '28px', '5xl': '32px' }, spacing: { 0.5: '4px', 1: '8px', 1.5: '12px', 2: '16px', 2.5: '20px', 3: '24px', 4: '32px', 4.5: '36px', 5: '40px', 6: '48px', 7: '56px', 8: '64px', 9: '72px', 10: '80px' } } } } Let’s go over what’s happening in this file. This first content block tells Tailwind where all of our email HTML templates and helpers live.{ content: ['app/helpers/mailer_helper.rb', 'app/views/*_mailer/*.html.erb', 'app/views/layouts/mailer.html.erb'] } Next, take a look at the future object:{ future: { disableColorOpacityUtilitiesByDefault: true } } This is the equivalent of saying:{ corePlugins: { backgroundOpacity: false, borderOpacity: false, divideOpacity: false, placeholderOpacity: false, ringOpacity: false, textOpacity: false } } And what this does if favor HEX values over RGBA values because, as you might have guessed, not all email clients support alpha values. Similarly, if you take a look at theme.extend in the tailwind.config.mailer.js file, this will favor PX values over REM, since email client doesn’t support them:{ theme: { extend: { // ... } } } Inline CSS Email clients don’t have great support for stylesheets. The easiest way to handle this is to work with inline styles, but that is error-prone and hard to work with, as you cannot use classes and/or reuse styling over your HTML. For our emails, we used a library called roadie to do the hard work for us. It also plays nice with Tailwind CSS. Add roadie-rails:./bin/bundle add roadie-rails Set up roadie-rails:# app/mailers/application_mailer.rb class ApplicationMailer < ActionMailer::Base + include Roadie::Rails::Automatic + default from: "from@example.com" layout "mailer" end Set up the layout The container is the main wrapper that hold your content. Typically, in email, this is a single ~600px wide center-aligned column that will shrink down on smaller viewports. Here is what ours looks like: <%= message.subject %> | PlanetScale <%= stylesheet_link_tag "mailer" %>
<%= yield %>
Supporting dark mode As part of these new emails, we had to make sure we support users who prefer dark mode. Tailwind made this a breeze: + + <%= message.subject %> | PlanetScale /* app/assets/stylesheets/mailer.css */ @import 'tailwindcss/base'; @import 'tailwindcss/components'; @import 'tailwindcss/utilities'; :root { color-scheme: light dark; supported-color-schemes: light dark; } @media (prefers-color-scheme: dark) { body { background-color: #111 !important; color: #fafafa !important; } a { color: #47b7f8 !important; } } Adding the preheader The preheader is the perfect place to further encourage your subscribers to open your email. It is the text headline that appears next to the email subject. Here’s how we set ours up:<% # app/views/database_mailer/database_weekly_report.html.erb %> <% content_for :preheader do %> <%= @report.period_start_label(full: false) %> – <%= @report.period_end_label(full: false) %> Here’s a look at the performance of your <%= @database.display_name %> database. <% end %> <% # app/views/layouts/mailer.html.erb %> <% if content_for?(:preheader) %> <% end %> Apple autolinking Phone numbers, addresses, dates, and (somewhat random) words like "tonight" frequently turn blue and underlined in emails viewed on an iPhone or iPad. These links trigger app-driven events, such as making a call or creating a calendar event. While these may come in handy for some scenarios, in others, they can be a nuisance and ruin your carefully-planned branding, even decreasing legibility. They weren’t relevant to our emails, so we removed them: + + <%= message.subject %> | PlanetScale <%= stylesheet_link_tag "mailer" %> Handling Gmail clipping Gmail clips emails that have a message size larger than 102KB and hides the content behind a “View entire message” link. Cut unnecessary content The first recommendation to handle this is to cut any content that may be unnecessary. In our case, we’re limiting the amount of slow queries to the first ten:-<% @report.slow_queries.each do |query| %> +<% @report.slow_queries.first(10).each do |query| %> - <%= query.sql %> + <%= truncate(query.sql, length: 200) %> Additionally you can truncate content and instead link to the full content:<% @report.slow_queries.first(10).each do |query| %> + <%= truncate(query.sql, length: 200 %> + + + <%= link_to "View query", query %> Debug view source After we went through and cut any unnecessary content, we noticed we’d still sometimes have clipped content. The email looks pretty small, so let’s take a closer look to see what sends the email size over the 102KB limit. To determine the size of your sent email, send it to a test address. View the source code, and save the source code in a document. Then, view the file size of that document. We were sending emails of ~80KB, so we wanted to create a bit more buffer. We did this by cutting down some Tailwind imports:-@import 'tailwindcss/base'; -@import 'tailwindcss/components'; @import 'tailwindcss/utilities'; @import 'mailer/base'; @import 'mailer/theme'; We removed Tailwind’s preflight styles and component utilities to try to bring down our overall file size:html { line-height: 1.5; } body { line-height: inherit; margin: 0; } img { border-style: none; display: block; vertical-align: middle; max-width: 100%; height: auto; } Testing file size again With these changes in place, we repeated the email sending test again, and saw we reduced our email size from 80KB to 45KB. It turned out that trying to accommodate Gmail’s clipping was a great exercise for determining what content is actually essential for your emails. Accommodating Gmail desktop styles We noticed that on desktop in Gmail, our mobile styles are applied, even though our design is responsive. According to the Google Workspace guides, Gmail supports CSS media queries. So, what’s going on here? Whenever you use responsive modifiers like sm or important modifiers like !, Tailwind CSS will escape that output. For example: This will generate:.hidden { display: none; } @media (min-width: 640px) { .sm\:\!inline { display: inline !important; } } While modern browsers support escaped sequences, Gmail unfortunately does not. It’s best to stick to the symbols [a-zA-Z0-9_-]. From a to z, from A to Z, from 0 to 9, underscores (_), and hyphens -. To do this, the easiest solution is to define a handful of utility helpers:@media (min-width: 640px) { .sm-block { display: block !important; } .sm-hidden { display: none !important; } .sm-inline { display: inline !important; } } And, finally, update our markup:- Sharding for cost-effective database management https://planetscale.com/blog/sharding-for-cost-effective-database-management 2023-11-13T13:00:00.000Z 2023-11-13T13:00:00.000Z David Bravant PlanetScale ranks 188th in Deloitte’s top 500 fastest-growing companies https://planetscale.com/blog/planetscale-named-deloitte-top-500-fastest-growing-companies 2023-11-08T09:00:00.000Z 2023-11-08T09:00:00.000Z Sam Lambert Announcing Vitess 18 https://planetscale.com/blog/announcing-vitess-18 2023-11-07T09:01:00.000Z 2023-11-07T09:01:00.000Z Vitess Engineering Team Announcing the Fivetran integration https://planetscale.com/blog/announcing-the-fivetran-integration 2023-11-02T12:00:00.000Z 2023-11-02T12:00:00.000Z Taylor Barnett Katie Sipos Introducing webhooks https://planetscale.com/blog/introducing-webhooks 2023-10-26T12:00:00.000Z 2023-10-26T12:00:00.000Z Taylor Barnett Mike Coutermarsh What is MySQL replication and when should you use it? https://planetscale.com/blog/what-is-mysql-replication-and-when-should-you-use-it 2023-10-25T15:00:00.000Z 2023-10-25T15:00:00.000Z Brian Morrison II Sync user data between Clerk and a PlanetScale MySQL database https://planetscale.com/blog/sync-user-data-between-clerk-and-a-planetscale-database 2023-10-20T15:00:00.000Z 2023-10-20T15:00:00.000Z Brian Morrison II "Email, Phone, Username" on the left side. Then click the "gear" next to "Email address". In the modal, enable the "Require" option and make sure "Email verification code" is enabled. Everything else should be disabled. Click "Continue" to accept the settings. Scroll down a bit and enable "Name", then click the "gear icon". In the modal, toggle "Require" to on and click "Continue". Scroll to the bottom of the page and click "Apply changes". Now you’ll need to grab the API keys for the project so the application will work when deployed. Select "API Keys" on the left and take note of both the "Publishable key" and the "Secret key". You’ll need these values as well when setting up Netlify. Anyone who has these values can access your Clerk project, so make sure to keep these values secret! Fork the project and deploy to Netlify Next, log into GitHub and fork the orbytal.ink project to your own account. Make sure "Copy the main branch only" is NOT enabled as you’ll want all branches forked into your account. Log into your Netlify account, and click "Add new site" > "Import an existing project". Select "Deploy with GitHub". Next, select "Orbytal.ink" from the list of projects in your account. Make sure to select the clerk-blog-post branch under "Branch to deploy". Then, scroll to the bottom and click "Deploy Orbytal.ink". Once the initial deployment is done, you’ll need to configure a few environment variables. Select "Site configuration" from the sidebar, and then "Environment variables". Add the following variables: DATABASE_URL: The connection string for your PlanetScale database formatted as mysql://:@aws.connect.psdb.cloud/orbytalink and replace DB_USERNAME and DB_PASSWORD with the values created earlier in the guide. CLERK_API_KEY: The private key acquired when configuring the Clerk project. VITE_CLERK_PUBLISHABLE_KEY: The public key acquired when configuring the Clerk project. Finally, re-deploy the project with the new settings by going back to "Deploys", and then "Trigger deploy" > "Deploy site". To test that your deployment works, navigate to the URL provided by Netlify and you should see the following if everything was built and deployed correctly. Add webhooks Now that your project is up and running, let's add the webhook so we can pass specific user information to the PlanetScale database as users are created or updated. Configure Clerk webhooks In the overview of the Clerk project, select "Webhooks" from the sidebar, and then "Add endpoint". Enter the "Endpoint URL" formatted as /.netlify/functions/clerk_webhook, select the topmost user element under "Message Filtering", and scroll to the bottom and click "Create". Create and deploy the Netlify Function Next, you’ll need to create a Netlify Function that will act as the endpoint to receive messages from our Clerk project. Luckily, all of this work can be done directly within GitHub. With the project open in GitHub, start by switching to the clerk-blog-post branch. Click "Add file" > "Create new file". Name the file functions/clerk_webhook.ts and paste the following into that file. Note the comments in the code that describe what the important parts of the function do.// functions/clerk_webhook.ts import { HandlerEvent, HandlerContext } from '@netlify/functions' import { getDb } from './utils/lib' import { blocks, users } from './utils/db/schema' import { eq } from 'drizzle-orm' // This type describes the structure of the incoming webhook type ClerkWebhook = { data: { first_name: string last_name: string image_url: string username: string } type: string } const handler = async (event: HandlerEvent, context: HandlerContext) => { if (event.body) { // 👉 Parse the incomign event body into a ClerkWebhook object const webhook = JSON.parse(event.body) as ClerkWebhook try { const db = getDb() // 👉 `webhook.type` is a string value that describes what kind of event we need to handle // 👉 If the type is "user.updated" the important values in the database will be updated in the users table if (webhook.type === 'user.updated') { await db .update(users) .set({ display_name: `${webhook.data.first_name} ${webhook.data.last_name}`, img_url: webhook.data.image_url }) .where(eq(users.username, webhook.data.username)) } // 👉 If the type is "user.created" create a record in the users table if (webhook.type === 'user.created') { await db.insert(users).values({ display_name: `${webhook.data.first_name} ${webhook.data.last_name}`, img_url: webhook.data.image_url, username: webhook.data.username }) } // 👉 If the type is "user.deleted", delete the user record and associated blocks if (webhook.type === 'user.deleted') { const dbuser = await db.query.users.findFirst({ where: eq(users.username, webhook.data.username) }) console.log('dbuser', dbuser) if (dbuser) { await Promise.all([ db.delete(users).where(eq(users.id, dbuser.id)), db.delete(blocks).where(eq(blocks.user_id, dbuser.id)) ]) } } return { statusCode: 200 } } catch (err) { console.error(err) return { statusCode: 500 } } } } export { handler } Click on "Commit changes" to open a modal. Feel free to add an extended description if you like, and click "Commit changes" in the modal to create the file in the branch. Once the file is saved, Netlify should automatically deploy the latest version of the web application. Test the function Now that the new code is deployed, we can test the three main operations that were configured in Clerk. Open the web app using the Netlify address and click "Create your profile". Create an account using your own email address. When prompted for a verification code, grab it from your email and enter it. If you get a message stating "The authentication settings are invalid", be sure to double check your configuration in Clerk. Once the account is created, you should be redirected back to your version of Orbytalink asking for some more details. Add a tagline and click "New block". Select "Twitter" and enter your Twitter username. Finally, click "Save". You should be redirected to your profile that not only shows you the tagline and Twitter block but also your name and username you entered into the Clerk sign-up form. That’s because when you created your account, Clerk sent a message to the Netlify Function you created earlier (clerk_webhook.ts) which saved this information into the PlanetScale database. If you explore the Netlify function that is used by the home page to retrieve user data, you’ll notice that there are no API calls to Clerk. It grabs information directly from the PlanetScale database and returns it to the React front end.// functions/profiles.ts import { HandlerEvent, HandlerContext } from '@netlify/functions' import { users } from './utils/db/schema' import { eq } from 'drizzle-orm' import { createResponse } from './utils/netlify_helpers' import { getDb } from './utils/lib' const handler = async (event: HandlerEvent, context: HandlerContext) => { const { username } = event.queryStringParameters as any const db = getDb() if (username) { const user = await db.query.users.findFirst({ where: eq(users.username, username), with: { blocks: true } }) return createResponse(200, user) } else { const user_rows = await db.select().from(users).limit(30) return createResponse(200, user_rows) } } export { handler } The following sequence diagram explains exactly how this overall system works: The user creates an account in Clerk. Clerk sends a message to the Netlify function once the user is created. Netlify writes that users’ info to the PlanetScale database. Using this flow, we can utilize Clerk to handle authentication and user management, and still have the users’ information available to us directly in our PlanetScale database! Conclusion Webhooks are extremely useful when using third-party systems where you need to be notified if a specific event happens in a place you don’t have full access to. In this guide, I showed you how you can use webhooks to receive user information from an IdP, but that’s only one example where these can be used in a production system. Have you used webhooks in your own projects? Let us know on Twitter by tagging @planetscale! If you enjoyed this article, you might also like our comprehensive guide on integrating AWS Lambda functions with PlanetScale.]]> Introducing database reports https://planetscale.com/blog/introducing-database-reports 2023-10-16T16:01:00.000Z 2023-10-16T16:01:00.000Z Frances Thai Distributed caching systems and MySQL https://planetscale.com/blog/distributed-caching-systems-and-mysql 2023-10-11T15:00:00.000Z 2023-10-11T15:00:00.000Z Brian Morrison II What is MySQL partitioning? https://planetscale.com/blog/what-is-mysql-partitioning 2023-10-10T15:00:00.000Z 2023-10-10T15:00:00.000Z Brian Morrison II MySQL High Availability: Connection handling and concurrency https://planetscale.com/blog/mysql-high-availability-connection-handling-concurrency 2023-10-10T13:00:00.000Z 2023-10-10T13:00:00.000Z Matthias Crauwels Personalizing your onboarding with Markdoc https://planetscale.com/blog/personalizing-your-onboarding-with-markdoc 2023-10-06T12:00:00.000Z 2023-10-06T12:00:00.000Z Ayrton {children} } Nodes in Markdoc To help contextualize the file that the onboarding code snippets live in, we want to extend the code blocks to accept an additional file attribute. Markdoc nodes enable you to customize how your document renders without using any custom syntax. The following example extends the code snippet from the previous section by adding a new fence node, which displays the filename above a code snippet.import React from 'react' import { parse, renderers, transform } from '@markdoc/markdoc' export default function Page() { const config = { nodes: { fence: Fence.scheme }, variables: { host: 'us-east.connect.psdb.cloud', user: 'mpl0y3jv3a92h4qc4ufn', database: 'beam', password: 'pscale_pw_V8db13jnq5mrOWcGFn6GTs6AerDI7A0womsmnJ1qxOc', ssl_ca: '/etc/ssl/certs/ca-certificates.crt' } } const doc = `# Configure your application\n…` const ast = parse(doc) const content = transform(ast, config) const children = renderers.react(content, React, { components: { Fence } }) return
{children}
} function Fence({ children, file, language }) { return (
{file}
        children
) } Fence.scheme = { render: Fence.name, children: ['pre', 'code'], attributes: { file: { type: String }, language: { type: String } } } Further customizations One challenge we have seen users face is deciding which SSL certificate to use when connecting securely to PlanetScale. To address this, we built a common component that will swap out the certificate based on the users detected operating system. This also extends the snippet from the previous section, adding in functions to detect the user's operating system and return the correct string for the ssl_ca variable.import React from 'react' import { parse, renderers, transform } from '@markdoc/markdoc' export default function Page({ userAgent }) { const platform = connectPlatform(userAgent) const sslCertificate = connectSslCertificate(platform) const config = { nodes: { fence: Fence.scheme }, variables: { host: 'us-east.connect.psdb.cloud', user: 'mpl0y3jv3a92h4qc4ufn', database: 'beam', password: 'pscale_pw_V8db13jnq5mrOWcGFn6GTs6AerDI7A0womsmnJ1qxOc', ssl_ca: sslCertificate } } const doc = `# Configure your application\n…` const ast = parse(doc) const content = transform(ast, config) const children = renderers.react(content, React, { components: { Fence } }) return
{children}
} function connectPlatform(userAgent) { userAgent = userAgent.toLowerCase() switch (true) { case /linux/.test(userAgent): return 'linux' case /mac/.test(userAgent): return 'mac' case /windows/.test(userAgent): return 'windows' default: return 'ubuntu' } } function connectSslCertificate(platform) { switch (platform) { case 'linux': return '/etc/ssl/certs/ca-certificates.crt' case 'mac': return '/etc/ssl/cert.pem' default: return '/etc/ssl/certs/ca-certificates.crt' } } function Fence({ children, file, language }) { return ( 
      
{file}


      children
) } Fence.scheme = { // … } Because a user's development environment is often different from their production environment, we also had to add a selector that allows users to select the certificate on their own. The final code for that is shown below.import React, { createContext, useState } from 'react' import { parse, renderers, transform } from '@markdoc/markdoc' const Platform = createContext({ platform: null, setPlatform: () => {} }) export default function Page({ userAgent }) { const initialPlatform = connectPlatform(userAgent) const [platform, setPlatform] = useState(initialPlatform) const sslCertificate = connectSslCertificate(platform) const config = { nodes: { fence: Fence.scheme }, variables: { host: 'us-east.connect.psdb.cloud', user: 'mpl0y3jv3a92h4qc4ufn', password: 'pscale_pw_V8db13jnq5mrOWcGFn6GTs6AerDI7A0womsmnJ1qxOc', ssl_ca: sslCertificate } } const doc = `# Configure your application\n…` const ast = parse(doc) const content = transform(ast, config) const children = renderers.react(content, React, { components: { Fence } }) return (
{children}
 ) } function connectPlatform(userAgent) { // … } function connectSslCertificate(platform) { // … } function Fence({ children, file, language }) { const { platform } = useContext(Platform) const isDotEnvFile = file === '.env' const isWindowsPlatform = platform === 'windows' return (
{file}
{isDotEnvFile && ( )} 
        children
{isDotEnvFile && (
{isWindowsPlatform && ( <> For Windows you may need to download a root certificate to connect securely.{' '} Learn more )} {!isWindowsPlatform && ( <> View the{' '} certificate authority root {' '} paths for the SSL CA details. )}
)}
) } Fence.scheme = { // … } Outcomes using Markdoc We are extremely happy with how the onboarding turned out, and based on some early data, it seems to be a huge win for new users as well. Working with Markdoc made the building process incredibly simple and straightforward. We're already finding that maintenance, like adding new frameworks, is very manageable as well. We'd love to hear if you've been able to get your hands on Markdoc yet. If you'd like to experience our onboarding process first-hand, make sure you sign up for a PlanetScale account to give it a go.]]> PlanetScale vs. Amazon Aurora https://planetscale.com/blog/planetscale-vs-amazon-aurora 2023-10-05T19:59:23.344Z 2023-10-05T19:59:23.344Z PlanetScale PlanetScale vs. Amazon RDS https://planetscale.com/blog/planetscale-vs-amazon-rds 2023-10-05T18:23:38.312Z 2023-10-05T18:23:38.312Z PlanetScale PlanetScale is bringing vector search and storage to MySQL https://planetscale.com/blog/planetscale-is-bringing-vector-search-and-storage-to-mysql 2023-10-03T09:00:00.000Z 2023-10-03T09:00:00.000Z Nick Van Wiggeren PlanetScale Managed is now PCI compliant https://planetscale.com/blog/planetscale-managed-is-now-pci-compliant 2023-10-02T09:00:00.000Z 2023-10-02T09:00:00.000Z Frank Fink Guide to scaling your database: When to shard MySQL and Postgres https://planetscale.com/blog/how-to-scale-your-database-and-when-to-shard-mysql 2023-09-28T08:00:00.000Z 2023-09-28T08:00:00.000Z Jonah Berquist Scaling hundreds of thousands of database clusters on Kubernetes https://planetscale.com/blog/scaling-hundreds-of-thousands-of-database-clusters-on-kubernetes 2023-09-27T15:00:00.000Z 2023-09-27T15:00:00.000Z Brian Morrison II The art and science of database sharding https://planetscale.com/blog/the-art-and-science-of-database-sharding 2023-09-19T13:00:00.000Z 2023-09-19T13:00:00.000Z Liz van Dijk Streamline database management using the PlanetScale Netlify integration https://planetscale.com/blog/planetscale-netlify-integration 2023-09-13T12:00:00.000Z 2023-09-13T12:00:00.000Z Taylor Barnett { const { planetscale: { connection } } = context const { body } = event if (!body) { return { statusCode: 400, body: 'Missing body' } } const { email, name } = JSON.parse(body) await connection.execute('INSERT INTO users (email, name) VALUES (?, ?)', [email, name]) return { statusCode: 201 } }) Getting started with the Netlify integration See the PlanetScale integration page in the Netlify docs to get started with the integration. You can also watch a past tech talk with PlanetScale and Netlify that uses the integration to learn more. If you have any suggestions for future features related to the integration, please let us know by tweeting at @planetscale or dropping us a note.]]> Emulating foreign key constraints with Drizzle relationships https://planetscale.com/blog/working-with-related-data-using-drizzle-and-planetscale 2023-09-06T17:30:00.000Z 2023-09-06T17:30:00.000Z Brian Morrison II users.id), label: varchar('label', { length: 200 }) }) Since Drizzle works across a number of different relational databases, using this method will automatically attempt to add foreign key constraints in the schema. Running the following command to apply this schema to a PlanetScale database using drizzle-kit results in an error:drizzle-kit push:mysql --schema functions/utils/db/schema.ts --connectionString='$DATABASE_URL' --driver mysql2 # Output: # Error: VT10001: foreign key constraints are not allowed [...] # { # code: 'ER_UNKNOWN_ERROR', # errno: 1105, # sql: 'ALTER TABLE `blocks` ADD CONSTRAINT `blocks_user_id_users_id_fk` FOREIGN KEY (`user_id`) REFERENCES `users`(`id`) ON DELETE no action ON UPDATE no action;', # sqlState: 'HY000', # sqlMessage: 'VT10001: foreign key constraints are not allowed' # } With just a bit more code, Drizzle can be configured to query the data in a child table using a virtual relationship instead of a foreign key constraint. The following code accomplishes the same results as above, allowing you to query a user and also get their associated blocks:export const users = mysqlTable('users', { id: serial('id').primaryKey(), username: varchar('username', { length: 120 }), tagline: varchar('tagline', { length: 250 }), display_name: varchar('display_name', { length: 250 }), img_url: varchar('img_url', { length: 500 }) }) export const blocks = mysqlTable('blocks', { id: serial('id').primaryKey(), url: varchar('url', { length: 200 }), block_type: int('type'), user_id: int('user_id'), label: varchar('label', { length: 200 }) }) //👇 This code block will tell Drizzle that users & blocks are related! export const usersRelations = relations(users, ({ many }) => ({ blocks: many(blocks) })) //👇 This code block defines which columns in the two tables are related export const blocksRelations = relations(blocks, ({ one }) => ({ user: one(users, { fields: [blocks.user_id], references: [users.id] }) })) Applying these changes using the same command as above will work as well.drizzle-kit push:mysql --schema functions/utils/db/schema.ts --connectionString='$DATABASE_URL' --driver mysql2 # Output: # drizzle-kit: v0.19.12 # drizzle-orm: v0.27.2 # # Reading schema files: orbytal-ink/functions/utils/db/schema.ts # # [✓] Changes applied Finally, when you want to return a user along with their associated blocks, you can use the following example:const user = await db.query.users.findFirst({ where: eq(users.username, username), // Providing `with` tells Drizzle you want to return related data with: { blocks: true } }) // Contents of `user`: // { // "id": 5, // "username": "brianmmdev", // "tagline": "Developer Educator @ PlanetScale", // "display_name": "Brian Morrison II", // "img_url": "https://img.clerk.com/eyJ0eXBlIjoicHJveHkiLCJzcmMiOiJodHRwczovL2ltYWdlcy5jbGVyay5kZXYvdXBsb2FkZWQvaW1nXzJUbzRXVjRkaFZRU0J2bTlxdnpsOXFiWWNyYS5qcGVnIn0", // "blocks": [ // { // "id": 9, // "url": "brianmmdev", // "block_type": 2, // "user_id": 5, // "label": null // }, // { // "id": 8, // "url": "brianmmdev", // "block_type": 4, // "user_id": 5, // "label": null // }, // { // "id": 7, // "url": "brianmmdev", // "block_type": 1, // "user_id": 5, // "label": null // } // ] // } What about cascading actions? One side effect available to foreign key constraints is cascading actions. Since PlanetScale does not support foreign key constraints, it’s not possible to specify these actions when designing your database schema. Luckily the solution is relatively straightforward. The responsibility shifts to the part you, as a developer, are likely most familiar with: the code. Earlier in this article, I suggested that foreign key constraints can be used to delete blocks associated with a user when that user is deleted. Below is the code that would need to be used to accomplish essentially the same thing:// This line will delete a user based on the passed in `userId` await db.delete(users).where(eq(users.id, userId)) // And this line will delete the associated blocks await db.delete(blocks).where(eq(blocks.user_id, userId)) As you can see, it’s only one more line of code that deletes the users’ blocks when that user is deleted. While this is definitely a simple example, you might ask “Doesn’t this require more work to accomplish the same thing?” Yes and no. It does indeed require more code on the part of the developer to maintain the integrity of the data within the database, however in a more complicated schema, you’ll likely have nested parent/child table relationships that can go several layers deep. If a topmost record is deleted, there is no guarantee that every single nested record will be able to be deleted since ALL constraints on the nested tables will need to be considered by the database engine. In this situation, the database may return an error that the developer will have to handle anyway, or worse yet the application will error out resulting in a poor user experience. By surfacing the task of maintaining the integrity of the data, you’re less likely to encounter these issues over time. Conclusion After reading this, you should be well-equipped on how to establish relationships using Drizzle without foreign key constraints. What are your thoughts on using Drizzle with PlanetScale? Let us know on Twitter and tag @planetscale!]]> Horizontal sharding for MySQL made easy https://planetscale.com/blog/horizontal-sharding-for-mysql-made-easy 2023-08-31T17:27:00.000Z 2023-08-31T17:27:00.000Z Lucy Burns Taylor Barnett Deploying multiple schema changes at once https://planetscale.com/blog/deploying-multiple-schema-changes-at-once 2023-08-29T09:00:00.000Z 2023-08-29T09:00:00.000Z Shlomi Noach What makes up a PlanetScale Vitess database? https://planetscale.com/blog/what-makes-up-a-planetscale-database 2023-08-23T15:00:00.000Z 2023-08-23T15:00:00.000Z Brian Morrison II Vitess for us all https://planetscale.com/blog/vitess-for-us-all 2023-08-22T13:00:00.000Z 2023-08-22T13:00:00.000Z Deepthi Sigireddi Introducing IP restrictions https://planetscale.com/blog/introducing-ip-restrictions 2023-08-15T12:00:00.000Z 2023-08-15T12:00:00.000Z Iheanyi Ekechukwu David Graham Ayrton Storing time series data in sharded MySQL to power Query Insights https://planetscale.com/blog/storing-time-series-data-in-sharded-mysql 2023-08-10T18:00:00.000Z 2023-08-10T18:00:00.000Z Rafer Hazen Is your database bleeding money? https://planetscale.com/blog/database-bleeding-money 2023-08-08T09:00:00.000Z 2023-08-08T09:00:00.000Z Sam Lambert How PlanetScale unlocks developer productivity https://planetscale.com/blog/how-planetscale-unlocks-developer-productivity 2023-07-26T14:01:00.000Z 2023-07-26T14:01:00.000Z Justin Gage Incorporating databases into your CI/CD pipeline https://planetscale.com/blog/databases-ci-cd-pipeline 2023-07-18T13:00:00.000Z 2023-07-18T13:00:00.000Z Mike Coutermarsh Performant database tree traversal with Rails https://planetscale.com/blog/performant-database-tree-traversal-with-rails 2023-07-12T17:30:00.000Z 2023-07-12T17:30:00.000Z Mike Coutermarsh Announcing PlanetScale Scaler Pro https://planetscale.com/blog/announcing-scaler-pro 2023-07-06T09:00:00.000Z 2023-07-06T09:00:00.000Z Nick Van Wiggeren Sharding vs. partitioning: What’s the difference? https://planetscale.com/blog/sharding-vs-partitioning-whats-the-difference 2023-06-30T17:10:05.128Z 2023-06-30T17:10:05.128Z PlanetScale Introduction to PlanetScale https://planetscale.com/blog/introduction-to-planetscale 2023-06-29T13:00:00.000Z 2023-06-29T13:00:00.000Z Taylor Barnett How PlanetScale keeps your data safe https://planetscale.com/blog/how-planetscale-keeps-your-data-safe 2023-06-28T00:03:57.138Z 2023-06-28T00:03:57.138Z Sam Lambert Announcing Vitess 17 https://planetscale.com/blog/announcing-vitess-17 2023-06-27T09:01:00.000Z 2023-06-27T09:01:00.000Z Matt Lord Action on your product data in real time https://planetscale.com/blog/action-on-your-product-data-in-real-time 2023-06-22T13:00:00.000Z 2023-06-22T13:00:00.000Z Brian Morrison II Datetimes versus timestamps in MySQL https://planetscale.com/blog/datetimes-vs-timestamps-in-mysql 2023-06-22T00:03:57.138Z 2023-06-22T00:03:57.138Z Aaron Francis Generated Hash Columns in MySQL https://planetscale.com/blog/generated-hash-columns-in-mysql 2023-06-15T00:03:57.138Z 2023-06-15T00:03:57.138Z Aaron Francis Using PlanetScale with Serverless Framework Node applications on AWS https://planetscale.com/blog/using-planetscale-with-serverless-framework-node-apps-on-aws 2023-06-13T17:03:57.138Z 2023-06-13T17:03:57.138Z Matthieu Napoli --secret Creating a new serverless Node application Serverless Framework is a CLI tool that helps us create and deploy serverless applications. Its configuration is stored in a serverless.yml file, which describes what will be deployed to AWS. To deploy a Node application, we can create a simple serverless.yml file:service: demo # name of the application provider: name: aws runtime: nodejs18.x region: us-east-1 functions: api: handler: index.handler url: true In the configuration above, we define a single AWS Lambda function called api, running NodeJS 18, with a public URL. Our API handler will be a handler() function returned by index.js (learn more about handlers in the AWS Lambda documentation). Let's create the index.js file:export async function handler(event) { return { hello: 'world' } } Note that we will be using ESM features (like export and import), so let's create a package.json file with "type": "module":{ "type": "module" } Our simple Node example is ready to be deployed with serverless deploy, but let's add PlanetScale into the mix first! Connecting to PlanetScale In your PlanetScale account, start by creating a database in the same region as the AWS application (us-east-1 in our example). Then, click the Connect button and select "Connect with: @planetscale/database". That will let us retrieve the database username and password. To connect to the database in our code, we will use the PlanetScale serverless driver. Let's install it with NPM:npm install @planetscale/database Now that the driver is installed, we can connect to our PlanetScale database with the connect() function:import { connect } from '@planetscale/database' const conn = connect({ // With the serverless driver, the host is always 'aws.connect.psdb.cloud' host: 'aws.connect.psdb.cloud', username: '', password: '' }) export async function handler(event) { const result = await conn.execute('SELECT * FROM records') return result.rows } Note the following details: We are connecting to the database outside the handler() function. This is to reuse the same connection for all HTTP requests. If we were to connect inside the handler() function, a new connection would be created for each request, which would be inefficient. We are querying the records table. This table doesn't exist yet, we will create it below. We don't want to store the database credentials in the code. We will use environment variables instead. Let's update our code to use environment variables. For the sake of the example, we will also create the records table on the fly:import { connect } from '@planetscale/database' const conn = connect({ // With the serverless driver, the host is always the same host: 'aws.connect.psdb.cloud', username: process.env.DATABASE_USERNAME, password: process.env.DATABASE_PASSWORD }) // Create the table if it doesn't exist (just for demo purposes) // In a real application, we would run database migrations outside the function await conn.execute('CREATE TABLE IF NOT EXISTS records (id INT PRIMARY KEY auto_increment, name VARCHAR(255))') export async function handler(event) { // Insert a new record const queryParameter = event.queryStringParameters?.name ?? 'test' await conn.execute('INSERT INTO records (name) VALUES (?)', [queryParameter]) // Retrieve all records const result = await conn.execute('SELECT * FROM records') return result.rows } We now need to set the DATABASE_USERNAME and DATABASE_PASSWORD environment variables. We can define them in serverless.yml and use AWS SSM to store the database password securely:provider: name: aws runtime: nodejs18.x region: us-east-1 environment: DATABASE_USERNAME: DATABASE_PASSWORD: ${ssm:/planetscale/db-password} The database password will be stored in AWS SSM (at no extra cost) so it is not visible in the code. The ${ssm:/planetscale/db-password} variable will retrieve the value from SSM on deployment. The SSM parameter can be created with the AWS CLI via the following command:aws ssm put-parameter --region us-east-1 --name '/planetscale/db-password' --type SecureString --value 'replace-me' # replace the `replace-me` string with the database password! If you don't use the AWS CLI, you can also create the parameter in the AWS Console: Our application is now ready! Let's deploy it:serverless deploy When finished, the deploy command will display the URL of our Node application. The URL should look like this: https://.lambda-url.us-east-1.on.aws/. We can open it in the browser or request it with curl:curl https://.lambda-url.us-east-1.on.aws/ The response should list the records in the records table. A new record will be created every time the URL is requested. We can also provide a name parameter to change the name of the record inserted in the database:curl https://.lambda-url.us-east-1.on.aws/?name=hello Stage parameters Besides the incredible scalability provided by the combination of AWS Lambda and PlanetScale, another benefit we get from this setup is the ability to combine Serverless Framework stages and PlanetScale branches. We could imagine, for example, a dev stage for development and a prod stage for production. The dev stage would use a development branch in PlanetScale, while the prod stage would use the main production branch. Using stage parameters, we can set different credentials to use to connect to PlanetScale depending on the stage:provider: name: aws runtime: nodejs18.x region: us-east-1 environment: DATABASE_USERNAME: ${param:dbUser} DATABASE_PASSWORD: ${param:dbPassword} params: dev: dbUser: dbPassword: ${ssm:/planetscale/dev/db-password} prod: dbUser: dbPassword: ${ssm:/planetscale/prod/db-password} When deploying, we can specify the stage to deploy via the --stage option:serverless deploy --stage dev serverless deploy --stage prod Each stage (dev and prod) will result in entirely separate infrastructures on AWS, and each one will use its own PlanetScale branch. That setup makes it easy to test code changes and database schema changes in a development environment that is identical to and isolated from the production environment. Once approved, schema changes can be applied to the production branch with a PlanetScale deploy request, and code changes can be deployed to production via the serverless deploy command. Next steps In this article, we learned how to integrate PlanetScale with Node applications built using the Serverless Framework on AWS. This gives us a completely serverless stack with extreme scalability yet simple to set up and maintain. Now that we have a basic application running we can explore more complex topics, such as: Creating multiple HTTP routes Setting up a complete deployment workflow for the Node application Dive into the PlanetScale workflow for branching databases, non-blocking schema changes, and more Feel free to explore the PlanetScale documentation as well as the Serverless Framework documentation to learn more.]]> PlanetScale joins AWS ISV Accelerate https://planetscale.com/blog/planetscale-joins-aws-isv-accelerate 2023-06-12T09:00:00.000Z 2023-06-12T09:00:00.000Z Nick Van Wiggeren Announcing the Hightouch integration https://planetscale.com/blog/announcing-the-hightouch-integration 2023-06-08T15:00:00.000Z 2023-06-08T15:00:00.000Z Brian Morrison II Using redundant conditions to unlock indexes in MySQL https://planetscale.com/blog/redundant-and-approximate-conditions 2023-06-07T00:03:57.138Z 2023-06-07T00:03:57.138Z Aaron Francis NOW() - INTERVAL 24 HOUR; -- | id | type | possible_keys | key | key_len | ref | rows | filtered | Extra | -- |----|-------|---------------|------------|---------|-----|------|----------|-----------------------| -- | 1 | range | created_at | created_at | 4 | | 1 | 100.00 | Using index condition | However, if we wrap this column in a function, we're obfuscating the column from MySQL, and it can no longer use the index.EXPLAIN SELECT * FROM todos WHERE YEAR(created_at) = 2023; -- | id | type | possible_keys | key | key_len | ref | rows | filtered | Extra | -- |----|------|---------------|-----|---------|-----|-------|----------|-------------| -- | 1 | ALL | | | | | 39746 | 100.00 | Using where | By wrapping the created_at column in a YEAR function, we're asking MySQL to do an index lookup on YEAR(created_at), which is not an index MySQL maintains. It is only maintaining the created_at index. In some cases, there are ways around index obfuscation. In this example, we could use a range scan instead of the YEAR function to obtain the same result.EXPLAIN SELECT * FROM todos WHERE created_at BETWEEN '2023-01-01 00:00:00' AND '2023-12-31 23:59:59'; -- | id | type | possible_keys | key | key_len | ref | rows | filtered | Extra | -- |----|-------|---------------|------------|---------|-----|------|----------|-----------------------| -- | 1 | range | created_at | created_at | 4 | | 1 | 100.00 | Using index condition | By unwrapping the created_at column and changing the comparison to a range scan, we've unlocked the index and allowed MySQL to use it effectively. Unfortunately, it's not always possible to de-obfuscate your indexes. In some scenarios, you simply cannot avoid wrapping the column in a function. In these cases, you might see if there is a redundant condition that could potentially unlock an existing index. Redundant conditions in MySQL A redundant condition is a condition that seems superfluous, extra, or not needed. It is a condition that can be added and removed without changing the results that MySQL returns. Let's take a look at a contrived example to illustrate the point. In this example, we're selecting the todos with an id of less than five.SELECT * FROM todos WHERE id < 5 In this case, a redundant condition might be id < 10.SELECT * FROM todos WHERE id < 5 and id < 10 -- This does... nothing This is a redundant condition because it does not change the results! Anything with an ID of less than five necessarily has an ID of less than ten also. You can add or remove this condition, and nothing will change. It's also silly to add because it doesn't provide us any benefit. We're going to expand our todos table definition a little bit to add due_date and due_time columns. (Storing date and time separately is usually not advised, but it helps us prove the point.)CREATE TABLE `todos` ( `id` int NOT NULL AUTO_INCREMENT, `title` varchar(255) NOT NULL, `due_date` date NOT NULL, `due_time` time NOT NULL, `created_at` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP, PRIMARY KEY (`id`), KEY `due_date` (`due_date`), KEY `created_at` (`created_at`) ) Given this table, if you want to query for todos that are due in the next day, you're stuck using the ADDTIME function:SELECT * FROM todos WHERE ADDTIME(due_date, due_time) BETWEEN NOW() AND NOW() + INTERVAL 1 DAY We do have an index on due_date, but the index cannot be used because we're performing an operation on it (adding the time). Unlike our previous example, there is no easy way to de-obfuscate this column either since the due_time is different for every row. We can confirm that the index is not being used by running an EXPLAIN on the previous query:| id | type | possible_keys | key | key_len | ref | rows | filtered | Extra | |----|------|---------------|-----|---------|-----|-------|----------|-------------| | 1 | ALL | | | | | 39746 | 100.00 | Using where | To work around this, let's add a redundant condition on due_date alone. When adding the condition, we need to make sure that it's logically impossible to change the result set, which means our redundant condition should be broader than our actual condition. Since we're looking for todos due in the next 24 hours, we can add a condition that looks for todos due today or tomorrow. That will contain the entire subset of todos that we're looking for and a few that we're not.EXPLAIN SELECT * FROM todos WHERE -- The real condition ADDTIME(due_date, due_time) BETWEEN NOW() AND NOW() + INTERVAL 1 DAY AND -- The redundant condition due_date BETWEEN CURRENT_DATE AND CURRENT_DATE + INTERVAL 1 DAY The redundant condition here returns a broader subset of todos than we need, but importantly it allows MySQL to use the index. Running an EXPLAIN on this query and we see that the due_date index was used:| id | type | possible_keys | key | key_len | ref | rows | filtered | Extra | |----|-------|---------------|----------|---------|-----|------|----------|------------------------------------| | 1 | range | due_date | due_date | 3 | | 1 | 100.00 | Using index condition; Using where | MySQL will first use the index to eliminate most of the table, then the slower ADDTIME will be used to eliminate the few remaining false positives. The redundant condition is doing its job perfectly! Domain-specific redundant conditions Until now, we've been working with redundant conditions that logically cannot change the result set. These are nice because they are easy to reason about and require no further domain knowledge. There are scenarios where you, as a human, might have more knowledge than the database does. (For now, at least.) In those situations, you might be able to add a redundant condition that is not logically incapable of changing the output, but you know, based on your knowledge, that it won't change the output. In the case of our todos table, let's add an updated_at column that will be populated with the timestamp of the last time the record was changed.CREATE TABLE `todos` ( `id` int NOT NULL AUTO_INCREMENT, `title` varchar(255) NOT NULL, `created_at` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP, `updated_at` timestamp DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP PRIMARY KEY (`id`), KEY `created_at` (`created_at`) ) In this scenario, we still only have an index on created_at, but if we want to query against updated_at, we might be able to add a redundant condition based on our knowledge of the application. If, given our understanding of the application, we can be sure that created_at is always equal to or earlier than updated_at, we can use this to our advantage. This query, which looks for records that were last modified before January 1st of 2023, will scan the entire table because there is no index on updated_at:SELECT * FROM todos WHERE updated_at < '2023-01-01 00:00:00' This query will return the same results but uses the created_at index to eliminate records and then filters out the false positives.SELECT * FROM todos WHERE updated_at < '2023-01-01 00:00:00' AND created_at < '2023-01-01 00:00:00' The only reason this redundant condition works is because we know that a record cannot be modified before it's created. Depending on your application, you might be able to find more examples of "domain-specific" redundant conditions. When to use a redundant condition The optimal indexing strategy always depends on the application, but in general, it's best to have indexes on the conditions you are frequently querying against. Redundant conditions are nice because they require no changes to the database! You can modify the query or the application generating the query, and suddenly everything gets faster. This makes them useful for queries that are only sometimes run or where indexes can't be easily added to the main conditions. If you'd like to learn more about indexing strategies, we have 17 videos on indexes as a part of our larger course on MySQL for Developers. If you do end up using the redundant condition strategy, please let us know on Twitter how you did it. We'd love to add more examples to this article!]]> Optimizing query planning in Vitess: a step-by-step approach https://planetscale.com/blog/optimizing-query-planning-in-vitess-a-step-by-step-approach 2023-06-01T15:00:00.000Z 2023-06-01T15:00:00.000Z Andres Taylor Pulling back the curtain: the new database overview page https://planetscale.com/blog/our-new-database-overview-page 2023-05-31T09:00:00.000Z 2023-05-31T09:00:00.000Z Holly Guevara Increase developer productivity with Database DevOps https://planetscale.com/blog/developer-productivity-database-devops 2023-05-25T13:00:00.000Z 2023-05-25T13:00:00.000Z Nick Van Wiggeren PlanetScale is now available on the Google Cloud Marketplace https://planetscale.com/blog/planetscale-is-now-available-on-the-google-cloud-marketplace 2023-05-22T09:00:00.000Z 2023-05-22T09:00:00.000Z Nick Van Wiggeren Character sets and collations in MySQL https://planetscale.com/blog/mysql-charsets-collations 2023-05-18T00:03:57.138Z 2023-05-18T00:03:57.138Z Aaron Francis table > database > server) is used. A collation can be defined at the column level, the table level, or it can be inherited from the character set default. Again, the most specific level is used. The character set and collation of a column affect how data is stored and how it is compared and sorted. Be mindful of these settings to ensure the correct behavior and optimal performance when designing your database. If you are unsure which character set or collation to use, the MySQL default utf8mb4 character set and its default utf8mb4_0900_ai_ci collation are usually good choices. They support all Unicode characters and provide case-insensitive and accent-insensitive comparisons.]]> MariaDB vs. MySQL https://planetscale.com/blog/mariadb-vs-mysql 2023-05-16T13:00:00.000Z 2023-05-16T13:00:00.000Z Matt Lord Backward compatible database changes https://planetscale.com/blog/backward-compatible-databases-changes 2023-05-09T16:10:00.000Z 2023-05-09T16:10:00.000Z Taylor Barnett Why isn’t MySQL using my index? https://planetscale.com/blog/why-isnt-mysql-using-my-index 2023-05-04T00:03:57.138Z 2023-05-04T00:03:57.138Z Aaron Francis Serverless Laravel applications with AWS Lambda and PlanetScale https://planetscale.com/blog/serverless-laravel-app-aws-lambda-bref-planetscale 2023-05-03T17:03:57.138Z 2023-05-03T17:03:57.138Z Matthieu Napoli --secret Getting started with Bref and Laravel Now that everything is ready, let's install Bref and its Laravel integration:composer require bref/bref bref/laravel-bridge --update-with-dependencies Then, let's create a serverless.yml configuration file:php artisan vendor:publish --tag=serverless-config This configuration file describes what will be deployed to AWS. Let's deploy now:serverless deploy When finished, the deploy command will display the URL of our Laravel application. Using PlanetScale as the database Now that Laravel is running in the cloud, let's set it up with a PlanetScale database. Start in PlanetScale by creating a new database in the same region as the AWS application (us-east-1 by default). Click the Connect button and select "Connect with: PHP (PDO)". That will let us retrieve the host, database name, user, and password. Edit the .env configuration file to set up the database connection:DB_CONNECTION=mysql DB_HOST= DB_PORT=3306 DB_DATABASE= DB_USERNAME= DB_PASSWORD= MYSQL_ATTR_SSL_CA=/opt/bref/ssl/cert.pem For DB_DATABASE, you can use your PlanetScale database name directly if you have a single unsharded keyspace. If you have a sharded keyspace, you'll need to use @primary. This will automatically direct incoming queries to the correct keyspace/shard. For more information, see the Targeting the correct keyspace documentation. Don't skip the MYSQL_ATTR_SSL_CA line: SSL certificates are required to secure the connection between Laravel and PlanetScale. Note that the path in AWS Lambda (/opt/bref/ssl/cert.pem) differs from the one on your machine (likely /etc/ssl/cert.pem). If you run the application locally, you will need to change this environment variable back and forth. Next, redeploy the application:serverless deploy Now that Laravel is configured, we can run database migrations to set up DB tables. To do so, we can run Laravel Artisan commands in AWS Lambda using the serverless bref:cli command:serverless bref:cli --args="migrate --force" That's it! Our database is ready to use. Creating sample data To test the database connection, let's create sample data in the users table created out of the box by Laravel. Edit the database/seeders/DatabaseSeeder.php class and uncomment the following line so that we can seed our database with 10 fake users: \App\Models\User::factory(10)->create(); Now, let's create a public API route that returns all the users from the database. Add the following code to routes/api.php:Route::get('/users', function () { return \App\Models\User::all(); }); Let's deploy these changes:serverless deploy Now, let's seed the database with 10 fake users:serverless bref:cli --args="migrate:fresh --seed --force" We can now retrieve our 10 users via the API route we created:curl https:///api/users Performance with a simple load test using PlanetScale The execution model of AWS Lambda gives us instant autoscaling without any configuration. To illustrate that, I have performed a simple load test against the application we deployed above. The only change I made is to disable Laravel's default rate limiting for API calls (ThrottleRequests middleware) in app/Http/Kernel.php because it would get in the way of my load test. Furthermore, I did not ramp up traffic progressively because I wanted to show Lambda's instant scalability. I used ab (Apache's benchmarking tool) to request the /api/users endpoint with 50 threads (50 HTTP requests made in parallel continuously):ab -c 50 -n 10000 https:///api/users When looking at the AWS Lambda and API Gateway metrics, we see the following numbers: Laravel scaled instantly from zero to 3,800 HTTP requests/minute. 100% of HTTP requests were handled successfully. The median PHP execution time (p50) for each HTTP request is 75ms. 95% of requests (p95) are processed in less than 130ms. PlanetScale processed up to 180 queries/s. The median PlanetScale query execution time is 0.3ms. The load test was performed against a freshly deployed application. That means the first requests were cold starts: New AWS Lambda instances started and scaled up to handle the incoming traffic. The cold starts usually have a much slower execution time (one second instead of 75ms). However, we do not see them in the p50 or p95 metrics because they only impacted 1% of the requests in the first minute. After the first 50 requests (cold starts), all the other requests were warm invocations. Note that we are looking at the AWS Lambda duration instead of HTTP response time: This is to exclude any latency related to networking (and thus have reproducible and comparable results). This is not the HTTP response time real users would see as, like on any server, the network adds latency to HTTP responses. After a few minutes, I dropped the traffic from 50 requests in parallel to one. The PHP execution time stayed identical. This illustrates that the load did not impact the response time. Improving performance to speed up the SSL connection For many web applications, responding in about 100ms is more than satisfactory. However, some use cases may require lower latency. Since Laravel connects to PlanetScale over SSL, creating the SSL connection can take longer than running the SQL query itself. PlanetScale itself can easily handle unlimited connections using built-in connection pooling, which massively improves performance by keeping those database connections open between requests. However, PHP, by design, shares nothing across requests. This means at the end of every request, PHP will close the connection to the database. To circumvent this problem, we can use Laravel Octane to gain performance in two ways: Keeping the Laravel application in memory across requests using Laravel Octane. Reusing SQL connections across requests (instead of reconnecting every time). Bref supports Laravel Octane natively. We need to change the serverless.yml configuration to enable it. Change the web function configuration to this:web: handler: Bref\LaravelBridge\Http\OctaneHandler runtime: php-81 environment: BREF_LOOP_MAX: 250 OCTANE_PERSIST_DATABASE_SESSIONS: 1 events: - httpApi: '*' Let's redeploy with serverless deploy and run the load test again: We notice the following improvements: The median PHP execution time (p50) went from 75ms to 14ms. 95% of requests (p95) are processed in less than 35ms. Laravel handled 1,000 more requests/minute, though this number is not important: We could simply send more requests in our load test to reach a higher number anytime. Going further and next steps Here are some next steps: Download and run the code used in this blog post. Learn more about using PlanetScale with Bref: MySQL compatibility, data imports, and schema changes workflow. Learn more about running Laravel on AWS Lambda: running Artisan commands, setting up assets, queues, and more. You can also learn more about PlanetScale and AWS Lambda with Bref in the respective documentation.]]> Database branching: three-way merge for schema changes https://planetscale.com/blog/database-branching-three-way-merge-schema-changes 2023-04-26T17:03:57.138Z 2023-04-26T17:03:57.138Z Shlomi Noach branch1. Likewise, compute diff2 as diff(main, branch2). Look at diff1(diff2(main)). If running diff1 over diff2(main) is invalid (examples to follow), there's a conflict. Likewise, attempt diff2(diff1(main)). If that's invalid, there's a conflict. If both are valid but diff1(diff2(main)) != diff2(diff1(main)), there is a conflict. If both are valid and diff1(diff2(main)) == diff2(diff1(main)), there is no conflict between the two branches. The algorithm is, in fact, more elaborate. But let's first walk through a few examples to understand how the diffs and three-way-merge work, and what SQL nuances we might hit. Example: no conflict Consider this simplified schema for the three branches:-- main: CREATE TABLE `customer` ( `id` int, PRIMARY KEY (`id`) ); -- branch1: CREATE TABLE `customer` ( `id` int, `name` varchar(255) NOT NULL DEFAULT '', PRIMARY KEY (`id`) ); -- branch2: CREATE TABLE `customer` ( `id` int, PRIMARY KEY (`id`) ); CREATE TABLE `delivery` ( `id` int, `customer_id` int, PRIMARY KEY (`id`) ); The diffs are:-- diff1: ALTER TABLE `customer` ADD COLUMN `name` varchar(255) NOT NULL DEFAULT '' -- diff2: CREATE TABLE `delivery` ( `id` int, `customer_id` int, PRIMARY KEY (`id`) ) Clearly, the two branches do not conflict with one another. One adds a column to customer, and the other creates delivery table. Applying the two diffs in either order ends up with the same end result:CREATE TABLE `customer` ( `id` int, `name` varchar(255) NOT NULL DEFAULT '', PRIMARY KEY (`id`) ); CREATE TABLE `delivery` ( `id` int, `customer_id` int, PRIMARY KEY (`id`) ); Example: clear conflict In the next example, both branches introduce a new column under the same name but with a different type:-- main: CREATE TABLE `customer` ( `id` int, PRIMARY KEY (`id`) ); -- branch1: CREATE TABLE `customer` ( `id` int, `subscription_type` enum('free', 'promotional', 'paid'), PRIMARY KEY (`id`) ); -- branch2: CREATE TABLE `customer` ( `id` int, `subscription_type` int unsigned NOT NULL DEFAULT 0, PRIMARY KEY (`id`) ); The diffs are:-- diff1: ALTER TABLE `customer` ADD COLUMN `subscription_type` enum('free', 'promotional', 'paid') -- diff2: ALTER TABLE `customer` ADD COLUMN `subscription_type` int unsigned NOT NULL DEFAULT 0 Clearly, applying both diffs on top of each other is destined to fail. You cannot add two columns under the same name. Example: subtle conflict How about adding two completely different columns?CREATE TABLE `customer` ( `id` int, `name` varchar(255) NOT NULL DEFAULT '', PRIMARY KEY (`id`) ); -- branch1: CREATE TABLE `customer` ( `id` int, `name` varchar(255) NOT NULL DEFAULT '', `subscription_type` enum('free', 'promotional', 'paid'), PRIMARY KEY (`id`) ); -- branch2: CREATE TABLE `customer` ( `id` int, `name` varchar(255) NOT NULL DEFAULT '', `joined_at` timestamp NOT NULL DEFAULT current_timestamp(), PRIMARY KEY (`id`) ); The diffs are:-- diff1: ALTER TABLE `customer` ADD COLUMN `subscription_type` enum('free', 'promotional', 'paid') -- diff2: ALTER TABLE `customer` ADD COLUMN `joined_at` timestamp NOT NULL DEFAULT current_timestamp() It's possible to apply both diffs, in any order. However, the resulting schema looks different depending on the order. It may look either:-- diff1(diff2(main)): CREATE TABLE `customer` ( `id` int, `name` varchar(255) NOT NULL DEFAULT '', `joined_at` timestamp NOT NULL DEFAULT current_timestamp(), `subscription_type` enum('free', 'promotional', 'paid'), PRIMARY KEY (`id`) ); -- diff2(diff1(main)): CREATE TABLE `customer` ( `id` int, `name` varchar(255) NOT NULL DEFAULT '', `subscription_type` enum('free', 'promotional', 'paid'), `joined_at` timestamp NOT NULL DEFAULT current_timestamp(), PRIMARY KEY (`id`) ); The order of columns in a table matters. Queries that run a SELECT * FROM customer and use positional arguments will get different columns at positions 3 and 4. The two branches conflict with each other. This is similar to a Git merge conflict where two branches append different rows to the end of a file. We could avoid the conflict if one of the branches positioned the new column anywhere but last. For example:CREATE TABLE `customer` ( `id` int, `subscription_type` enum('free', 'promotional', 'paid'), `name` varchar(255) NOT NULL DEFAULT '', PRIMARY KEY (`id`) ); The above would lead to a non-conflicting diff:-- diff1: ALTER TABLE `customer` ADD COLUMN `subscription_type` enum('free', 'promotional', 'paid') AFTER `id` Nuance: no conflict The same cannot be said for index changes. We now add a column and a matching index in one migration, and another index in the second migration:-- main: CREATE TABLE `customer` ( `id` int, `name` varchar(255) NOT NULL DEFAULT '', PRIMARY KEY (`id`) ); -- branch1: CREATE TABLE `customer` ( `id` int, `name` varchar(255) NOT NULL DEFAULT '', PRIMARY KEY (`id`), KEY `name_idx` (`name`(16)) ); -- branch2: CREATE TABLE `customer` ( `id` int, `name` varchar(255) NOT NULL DEFAULT '', `joined_at` timestamp NOT NULL DEFAULT current_timestamp(), PRIMARY KEY (`id`), KEY `joined_idx` (`joined_at`) ); The diffs are:-- diff1: ALTER TABLE `customer` ADD KEY `name_idx` (`name`(16)) -- diff2: ALTER TABLE `customer` ADD COLUMN `joined_at` timestamp NOT NULL DEFAULT current_timestamp(), ADD KEY `joined_idx` (`joined_at`) Strictly speaking, the table structure looks different based on the order we apply the diffs. It can be either of:-- diff1(diff2(main)): CREATE TABLE `customer` ( `id` int, `name` varchar(255) NOT NULL DEFAULT '', `joined_at` timestamp NOT NULL DEFAULT current_timestamp(), PRIMARY KEY (`id`), KEY `joined_idx` (`joined_at`), KEY `name_idx` (`name`(16)) ); -- diff2(diff1(main)): CREATE TABLE `customer` ( `id` int, `name` varchar(255) NOT NULL DEFAULT '', `joined_at` timestamp NOT NULL DEFAULT current_timestamp(), PRIMARY KEY (`id`), KEY `name_idx` (`name`(16)), KEY `joined_idx` (`joined_at`) ); However, for practical purposes, the order of indexes is inconsequential. All queries against the table will both behave in the exact same way, as well as perform in the same way, irrespective of the ordering of the keys. The only change is the output of SHOW CREATE TABLE as well as INFORMATION_SCHEMA introspection. PlanetScale disregards index ordering. Overlapping changes The algorithm is more elaborate than described thus far. To reduce developer friction as much as possible, it also considers identical, partial overlap between diffs. For example:-- main: CREATE TABLE `customer` ( `id` int, PRIMARY KEY (`id`) ); -- branch1: CREATE TABLE `customer` ( `id` int, `name` varchar(255) NOT NULL DEFAULT '', PRIMARY KEY (`id`) ); CREATE TABLE `tbl1` ( `id` int, PRIMARY KEY (`id`) ); -- branch2: CREATE TABLE `customer` ( `id` int, `name` varchar(255) NOT NULL DEFAULT '', PRIMARY KEY (`id`) ); CREATE TABLE `tbl2` ( `id` int, PRIMARY KEY (`id`) ); Both branches create the same name column on customer, and then each branch proceeds to make other unrelated changes. Thanks to schemadiff, each of the changes (ALTER, CREATE, ...) is fully formalized and we can analyze the changes one by one. Is there a conflict with the new name column? Given that both branches completely agree on that particular change, PlanetScale's three-way merge considers this as an overlap and allows it. Should branch1 merge first, branch2's diff auto-adapts and is left to the creation of tbl2 only. Further reducing friction Schema changes may take time to run, during which more developers will want to deploy their own changes. There is a deployment queue, first come first served, that only allows a single deploy request at a time to run. When a developer submits their deploy request, their change is validated against all queued changes. This avoids the situation where the developer waits for hours in queue, only to learn the one deployment before theirs caused a conflict. PlanetScale shoots an early warning so that developers can better use their time in queue. Conclusion Schema changes and source code changes share enough similarities that we can offer developers schema lifecycle workflows they are familiar with from their source code workflows. With some adaptations to the obvious differences and challenges a schema change deployment poses, we are able to utilize familiar and trusted logic to manage developer collaboration around schema branching.]]> Query performance analysis with Insights https://planetscale.com/blog/query-performance-analysis-with-insights 2023-04-20T12:00:00.000Z 2023-04-20T12:00:00.000Z Rafer Hazen MySQL for application developers https://planetscale.com/blog/mysql-application-developers 2023-04-20T00:03:57.138Z 2023-04-20T00:03:57.138Z Aaron Francis Pagination in MySQL https://planetscale.com/blog/mysql-pagination 2023-04-18T00:03:57.138Z 2023-04-18T00:03:57.138Z Aaron Francis 10 -- The last id that the user saw was 10, so we start at the next id after 10 ORDER BY id LIMIT 10 You can see that in this query, we're not using the OFFSET keyword at all, but instead, we're jumping straight to the next record after the last record that the user saw. This is the key difference between cursor and offset-based pagination! It gets a bit more complicated if we go back to our original example of sorting by first_name and then id. Since we're sorting by both columns, the cursor must contain both values for the last record that the user has seen. Let's take this example set of records, which is 20 people sorted by first name, and then ID.| id | first_name | last_name | |-------|------------|------------| | 2 | Aaron | Francis | | 589 | Aaron | Streich | | 3896 | Aaron | Corkery | | 8441 | Aaron | Kreiger | | 9179 | Aaron | Wolf | | 10970 | Aaron | Reichert | | 13082 | Aaron | Collier | | 13704 | Aaron | Braun | | 19399 | Aaron | Watsica | | 25995 | Aaron | Runte | |-------|------------|------------| Page break | 26794 | Aaron | Mayer | | 32075 | Aaron | Hahn | | 32471 | Aaron | Bahringer | | 40612 | Aaron | Abbott | | 41202 | Aaron | Willms | | 41571 | Aaron | Nienow | | 46556 | Aaron | Glover | | 48501 | Aaron | Boyle | | 50628 | Aaron | Schmeler | | 51656 | Aaron | Williamson | In this case, the last record the user sees on page 1 has an id of 25995. This information alone is not enough for the cursor! We must also add the first_name since it is part of the sort order. The cursor for the last record on page 1 is (first_name=Aaron, id=25995). When the user sends back the cursor, we can construct a WHERE clause that filters out all the rows the user has already seen. This time, it requires a little more thought because we're sorting by two columns. We'll add a first_name filter to show any names after "Aaron," but since first_name has many duplicates, we'll also add an id filter to show any "Aaron"s that have an id after the last id that the user saw.SELECT * FROM people WHERE ( (first_name > 'Aaron') -- Names after Aaron OR (first_name = 'Aaron' AND id > 25995) -- Aarons, but after the last id that the user saw ) ORDER BY first_name, id LIMIT 10 As you add more columns to the sort order, you'll need to add more filters to the WHERE clause. Drawbacks to cursor-based pagination As you've seen, cursor-based pagination is more complicated to implement than offset-based pagination. Constructing the cursor and the WHERE clause requires more thought. You also have to keep track of that little piece of state: the cursor. This isn't inherently bad, and not all complexity is reducible, but it's something to keep in mind. Most frameworks have cursor-based pagination built in, so you may not have to implement it manually. Another drawback to cursor-based pagination is that it's impossible to address a specific page directly. For instance, if the requirement is to jump directly to page five, it's not possible to do so since the pages themselves are not explicitly numbered, and there is no way to create a cursor without knowing the last record that has been seen. You can only navigate to the next page. Benefits of cursor-based pagination One of the advantages of cursor-based pagination is its resilience to shifting rows. For example, if a record is deleted, the next record that would have followed is still displayed since the query is working off of the cursor rather than a specific offset. Let's go back to our Sonya Dickens example. The last person they see on this page is "Judge Bins." They don't see her yet, but "Sonya Dickens" should be the first person on page 2.| id | first_name | last_name | |----|------------|-----------| | 1 | Phillip | Yundt | | 2 | Aaron | Francis | | 3 | Amelia | West | | 4 | Jennifer | Becker | | 5 | Macy | Lind | | 6 | Simon | Lueilwitz | | 7 | Tyler | Cummerata | | 8 | Suzanne | Skiles | | 9 | Zoe | Hill | | 10 | Judge | Bins | <-- The cursor points here |----|------------|-----------| Page break | 11 | Sonya | Dickens | | 12 | Hope | Streich | | 13 | Kristian | Kerluke | | 14 | Stanton | Fisher | | 15 | Rasheed | Little | | 16 | Deron | Koss | | 17 | Trevor | Daniel | | 18 | Vernie | Friesen | | 19 | Jody | Littel | | 20 | Jorge | Nienow | While they are viewing page one, "Aaron Francis" is deleted.| id | first_name | last_name | |----|------------|-----------| | 1 | Phillip | Yundt | | 3 | Amelia | West | <-- Aaron Francis is deleted | 4 | Jennifer | Becker | | 5 | Macy | Lind | | 6 | Simon | Lueilwitz | | 7 | Tyler | Cummerata | | 8 | Suzanne | Skiles | | 9 | Zoe | Hill | | 10 | Judge | Bins | <-- The cursor *still* points here |----|------------|-----------| Page break | 11 | Sonya | Dickens | <-- Sonya is the first person after the cursor | 12 | Hope | Streich | | 13 | Kristian | Kerluke | | 14 | Stanton | Fisher | | 15 | Rasheed | Little | | 16 | Deron | Koss | | 17 | Trevor | Daniel | | 18 | Vernie | Friesen | | 19 | Jody | Littel | | 20 | Jorge | Nienow | This time, it doesn't matter! The cursor points to the last record that the user saw, and the next record is still Sonya Dickens. We tell the database, "the last record I saw was ID 10, and I want to see the next ten records." The database doesn't care that some records were deleted. It just knows that the next record is Sonya Dickens. This is true even if the cursor is pointing to a record that was deleted. If the cursor points to a record that was deleted, we're still telling the database, "the last record I saw was ID 10, and I want to see the next ten records." Again, the database doesn't care that the record was deleted. It just knows that the next record is Sonya Dickens. Cursor based pagination performance Cursor-based pagination can be much more performant than offset/limit simply because it accesses much less data. Instead of generating a result set and throwing away everything before the offset, the database can start at the offset and return the next N records. This is especially true if the offset is large. You will need to consider a proper indexing strategy to ensure the database can efficiently find the necessary records. Conclusion Pagination is a common requirement for almost every web application or API. Now you understand the different types of pagination and the tradeoffs that come with each. Offset/limit is nice because it's easy to implement and understand, and you can directly address pages. Some downsides are that it can be slower as you navigate deeper into the pages, and it is more prone to drift. Cursor-based pagination is nice because it is more performant and more resilient to shifting rows. Some of the downsides are that it is more complicated to implement, and you cannot directly address pages. Which method you choose is up to you, but hopefully, this article has given you a better understanding of the tradeoffs, and you can now make an informed decision.]]> Safely making database schema changes https://planetscale.com/blog/safely-making-database-schema-changes 2023-04-13T14:00:00.000Z 2023-04-13T14:00:00.000Z Taylor Barnett What is database sharding and how does it work? https://planetscale.com/blog/what-is-database-sharding-and-how-does-it-work 2023-04-06T09:00:00.000Z 2023-04-06T09:00:00.000Z Justin Gage An update to our workflow: safe migrations https://planetscale.com/blog/update-to-our-workflow-safe-migrations 2023-04-05T15:50:00.000Z 2023-04-05T15:50:00.000Z Nick Van Wiggeren Declarative schema migrations https://planetscale.com/blog/declarative-schema-changes 2023-04-05T14:00:00.000Z 2023-04-05T14:00:00.000Z Brian Morrison II sam-go-sample Resources: HelloWorldFunction: Type: AWS::Serverless::Function Properties: CodeUri: hello-world/ Handler: hello-world Runtime: go1.x Events: CatchAll: Type: Api Properties: Path: /hello Method: GET Performing the above actions manually, while not prohibitively difficult, would certainly take more time than deploying this configuration with a simple CLI command. This is also a fairly simple example. Consider how much manual effort it would take to configure and deploy 20 Lambda functions! Declarative SQL Schemas Several tools can manage your database schema in a very similar way to IaC tools. Using these tools, you can define your SQL schema in a specially-crafted file that the tool can understand, and simply apply the changes using the CLI. For example, the following file can be used by the Atlas CLI to define a schema:table "hotels" { schema = schema.hotels_db column "id" { null = false type = int unsigned = true auto_increment = true } column "name" { null = false type = varchar(50) } column "address" { null = false type = varchar(50) } primary_key { columns = [column.id] } } schema "hotels_db" { charset = "utf8mb4" collate = "utf8mb4_0900_ai_ci" } Making a change to the schema is as simple as modifying the file and applying the changes using the CLI tool.table "hotels" { schema = schema.hotels_db column "id" { null = false type = int unsigned = true auto_increment = true } column "name" { null = false type = varchar(50) } column "address" { null = false type = varchar(50) } # Adding the "stars" column. column "stars" { null = true type = float unsigned = true } primary_key { columns = [column.id] } } schema "hotels_db" { charset = "utf8mb4" collate = "utf8mb4_0900_ai_ci" } Refer to our blog post on how to use the Atlas CLI with PlanetScale for more detail. Benefits of a declarative approach Managing schema migrations with this approach has some benefits. The first major benefit is that it fits the Single Source of Truth approach encouraged by DevOps, where there is one place that contains the main file used to control the schema. It is also easier to read by developers in comparison to using versioned migrations. In addition to being easier to understand, it may eliminate the need to learn DDL, the language used by SQL to define the schema. This makes it a lower barrier to entry for developers that may not be experienced with SQL yet. Finally, automating the process of applying changes is fairly simple since many of the tools used to apply changes can be scripted. This makes it easy to implement the process of upgrading your schema into your continuous deployment tools. Drawbacks of this strategy While eliminating the need to learn DDL can be a benefit, using tools to circumvent the process of learning may act as a crutch for developers. Conflicting schema definitions are also a concern with this approach. If you consider that multiple developers may be making changes to the schema definition files at the same time on separate machines, you may run into a scenario where one developer's changes will overwrite another's, causing conflicts in what the database schema should be. It’s also worth considering that databases are inherently stateful, where the data that is stored by the database is just as important as the structure of the database. Because of this, some care needs to be taken when applying changes so there are no undesired results of migrating the schema. How to use declarative migrations with PlanetScale The branching flow used by databases hosted in PlanetScale is a form of schema migration in itself. When making changes to a database in PlanetScale, developers will typically create a working branch of the production database branch to make changes to. A best practice on PlanetScale is to enable safe migrations to prevent accidental changes to your database schema. Since these branches restrict the use of DDL (something that these tools ultimately use to make changes), the development branch used in the previous example would be where these tools can be used to control the schema. One possible strategy that teams can use is to open a new branch each time code changes are required, typically at the beginning of a development cycle. When a change needs to be made to the database schema, a dedicated repository (let’s call it the db repository) can be used for developers to check in changes to the definition file. Automated tools can be used to monitor the db repository for changes, apply the schema changes to the active development branch, and notify the development team that the schema has changed so they can act accordingly. When changes need to be applied to the production database branch, deploy requests can then be used to review and apply the changes before deploying the latest release.]]> Versioned schema migrations https://planetscale.com/blog/versioned-schema-migrations 2023-04-05T14:00:00.000Z 2023-04-05T14:00:00.000Z Brian Morrison II id(); $table->string('name'); $table->string('email')->unique(); $table->timestamp('email_verified_at')->nullable(); $table->string('password'); $table->rememberToken(); $table->timestamps(); }); } public function down() { Schema::dropIfExists('users'); } }; To create the basic structure of the database, the following command will be run. Notice how ALL migration scripts within that folder are run sequentially based on the file name.~❯ ./vendor/bin/sail artisan migrate # Output: INFO Preparing database. Creating migration table .............................. 45ms DONE INFO Running migrations. 2014_10_12_000000_create_users_table .................. 45ms DONE 2014_10_12_100000_create_password_resets_table ........ 64ms DONE 2019_08_19_000000_create_failed_jobs_table ............ 38ms DONE 2019_12_14_000001_create_personal_access_tokens_table . 44ms DONE Next, we can explore the structure of the database. Notice how a migrations table exists now and it contains the name of each of the migration scripts, along with a batch number stored in the batch column to signal to artisan that it's been run previously.mysql> show tables; +------------------------+ | Tables_in_example_app | +------------------------+ | failed_jobs | | migrations | | password_resets | | personal_access_tokens | | users | +------------------------+ 5 rows in set (0.01 sec) mysql> select * from migrations; +----+-------------------------------------------------------+-------+ | id | migration | batch | +----+-------------------------------------------------------+-------+ | 1 | 2014_10_12_000000_create_users_table | 1 | | 2 | 2014_10_12_100000_create_password_resets_table | 1 | | 3 | 2019_08_19_000000_create_failed_jobs_table | 1 | | 4 | 2019_12_14_000001_create_personal_access_tokens_table | 1 | +----+-------------------------------------------------------+-------+ 4 rows in set (0.01 sec) Now to upgrade the schema, we can run another migration script that follows the same naming convention as the others. This script will add a nickname column to the users table.# 2023_01_13_000001_add_new_column.php string('nickname'); }); } public function down() { Schema::table('users', function (Blueprint $table) { $table->dropColumn('nickname'); }); } }; Now we'll run the same migrate command as was run before. The output will be much less since it is only the one script that is run.~❯ ./vendor/bin/sail artisan migrate INFO Running migrations. 2023_01_13_000001_add_new_column ...................... 32ms DONE Reviewing the migrations table again shows that the script was run successfully.mysql> select * from migrations; +----+-------------------------------------------------------+-------+ | id | migration | batch | +----+-------------------------------------------------------+-------+ | 1 | 2014_10_12_000000_create_users_table | 1 | | 2 | 2014_10_12_100000_create_password_resets_table | 1 | | 3 | 2019_08_19_000000_create_failed_jobs_table | 1 | | 4 | 2019_12_14_000001_create_personal_access_tokens_table | 1 | | 5 | 2023_01_13_000001_add_new_column | 2 | +----+-------------------------------------------------------+-------+ And if we inspect the users table, the nickname column now exists.mysql> describe users; +-------------------+-----------------+------+-----+---------+----------------+ | Field | Type | Null | Key | Default | Extra | +-------------------+-----------------+------+-----+---------+----------------+ | id | bigint unsigned | NO | PRI | NULL | auto_increment | | name | varchar(255) | NO | | NULL | | | email | varchar(255) | NO | UNI | NULL | | | email_verified_at | timestamp | YES | | NULL | | | password | varchar(255) | NO | | NULL | | | remember_token | varchar(100) | YES | | NULL | | | created_at | timestamp | YES | | NULL | | | updated_at | timestamp | YES | | NULL | | | nickname | varchar(255) | NO | | NULL | | +-------------------+-----------------+------+-----+---------+----------------+ Now if I wanted to undo the previous migration for whatever reason, the following command can be run to essentially execute the down() function from the previous migration.~ ❯ ./vendor/bin/sail artisan migrate:rollback --step=1 INFO Rolling back migrations. 2023_01_13_000001_add_new_column ...................... 41ms DONE Reviewing the same tables one more time shows that the column has now been removed.mysql> select * from migrations; +----+-------------------------------------------------------+-------+ | id | migration | batch | +----+-------------------------------------------------------+-------+ | 1 | 2014_10_12_000000_create_users_table | 1 | | 2 | 2014_10_12_100000_create_password_resets_table | 1 | | 3 | 2019_08_19_000000_create_failed_jobs_table | 1 | | 4 | 2019_12_14_000001_create_personal_access_tokens_table | 1 | +----+-------------------------------------------------------+-------+ 4 rows in set (0.01 sec) mysql> describe users; +-------------------+-----------------+------+-----+---------+----------------+ | Field | Type | Null | Key | Default | Extra | +-------------------+-----------------+------+-----+---------+----------------+ | id | bigint unsigned | NO | PRI | NULL | auto_increment | | name | varchar(255) | NO | | NULL | | | email | varchar(255) | NO | UNI | NULL | | | email_verified_at | timestamp | YES | | NULL | | | password | varchar(255) | NO | | NULL | | | remember_token | varchar(100) | YES | | NULL | | | created_at | timestamp | YES | | NULL | | | updated_at | timestamp | YES | | NULL | | +-------------------+-----------------+------+-----+---------+----------------+ 8 rows in set (0.01 sec) Benefits of this strategy As stated in the previous section, versioned schema migrations have been around for much longer than declarative migrations. This means developers are likely more familiar with how they work and may be more comfortable working in this environment. Many tools that support versioned migrations support going both directions, upgrading and/or downgrading the schema. This makes reverting changes simpler since a single script will have instructions on performing a downgrade, assuming the developers or database administrators include those details in the migration scripts. Finally, it's easier to track incremental changes without using a version control system. Since all of the migration scripts are stored alongside each other, diagnosing migration issues may be a bit more straightforward when compared to the declarative approach. Drawbacks of this strategy Since the schema is managed incrementally via scripts, it may be hard to get a full picture of what the database schema looks like at any given point in time. You’d essentially have to replay all of the previous scripts against a live system to see the schema in full. Depending on the tool, it may not validate the current state of the schema before attempting to apply changes. This can cause major issues if the schema was modified outside of the tool and DDL was issued directly to the database. How to use versioned schema migrations with PlanetScale How you would use versioned migrations on PlanetScale ultimately depends on if safe migrations is enabled for your production branch. Without safe migrations If safe migrations is not enabled for your production branch, versioned migrations would work with PlanetScale branches just as they would with any other MySQL environment. Ideally, you would use different database branches to match your different environments. When your code is ready for production, simply run the upgrade command for your respective migration tools with the connection string for the branch you want the changes to, and your tooling should apply the changes. That said, enabling safe migrations is a best practice to prevent unintended schema changes, among enabling other useful features. With safe migrations When safe migrations is enabled on your production branch, use of branching and deploy requests is enforced to enable zero-downtime migrations, and use of direct DDL is restricted as a result. In this scenario, you would create a development branch, connect your development environment to the PlanetScale development branch, and run your migrations there. Your development branch will now have the updated schema, and is ready to merge into your production database via a PlanetScale deploy request. Typically when deploy requests are used to merge database branches, it's only the schema that is changed in the target without writing or altering any data. While this may seem like an issue at first (since a table is used to track what changes have been applied), PlanetScale offers a setting in every Vitess database to automatically copy migration data between branches. This can be set to several preconfigured ORMs, or you can provide a custom table name to sync between database branches. For additional examples of handling versioned schema changes with PlanetScale, see the following blog posts: Building PlanetScale with PlanetScale Zero downtime Laravel migrations ]]> Announcing the PlanetScale GitHub Actions https://planetscale.com/blog/announcing-the-planetscale-github-actions 2023-03-31T09:00:00.000Z 2023-03-31T09:00:00.000Z Brian Morrison II > $GITHUB_ENV # Create the DATABASE_URL database_url="mysql://$username:$password@$host/${{ secrets.PLANETSCALE_DATABASE_NAME }}?sslmode=$ssl_mode&sslca=$ssl_ca" echo "DATABASE_URL=$database_url" >> $GITHUB_ENV echo "::add-mask::$DATABASE_URL" - name: Use the DATABASE_URL in a subsequent step run: | echo "Using DATABASE_URL: $DATABASE_URL" This example shows creating the password and getting back a response in JSON. The JSON is then parsed to create a DATABASE_URL which can be used in later steps, such as usingthe branch as the database for a preview environment or to connect and run migrations that were included in the GitHub pull request. Open a deploy request You can use pscale deploy-request create to open a new deploy request from GitHub Actions.This can be useful after running migrations against a branch.- name: Open DR if migrations env: PLANETSCALE_SERVICE_TOKEN_ID: ${{ secrets.PLANETSCALE_SERVICE_TOKEN_ID }} PLANETSCALE_SERVICE_TOKEN: ${{ secrets.PLANETSCALE_SERVICE_TOKEN }} run: pscale deploy-request create ${{ secrets.PLANETSCALE_DATABASE_NAME }} ${{ env.PSCALE_BRANCH_NAME }} Get deploy request diff and comment on pull request We can use pscale deploy-request diff to see the full schema diff of a deploy request. This example is useful when combined with opening a deploy request for a git branch. You can then automatically comment the diff back to the GitHub pull request.- name: Comment on PR env: PLANETSCALE_SERVICE_TOKEN_ID: ${{ secrets.PLANETSCALE_SERVICE_TOKEN_ID }} PLANETSCALE_SERVICE_TOKEN: ${{ secrets.PLANETSCALE_SERVICE_TOKEN }} run: | echo "Deploy request opened: https://app.planetscale.com/${{ secrets.PLANETSCALE_ORG_NAME }}/${{ secrets.PLANETSCALE_DATABASE_NAME }}/deploy-requests/${{ env.DEPLOY_REQUEST_NUMBER }}" >> migration-message.txt echo "" >> migration-message.txt echo "\`\`\`diff" >> migration-message.txt pscale deploy-request diff ${{ secrets.PLANETSCALE_DATABASE_NAME }} ${{ env.DEPLOY_REQUEST_NUMBER }} -f json | jq -r '.[].raw' >> migration-message.txt echo "\`\`\`" >> migration-message.txt - name: Comment PR - db migrated uses: thollander/actions-comment-pull-request@v2 with: filePath: migration-message.txt This writes the diff to the migration-message.txt file and then creates a comment on the pull request that triggered the workflow. How to use the PlanetScale GitHub Actions To get started using PlanetScale + GitHub Actions, see our full guide and complete page of examples here.]]> Building SaaS applications with PlanetScale + Netlify https://planetscale.com/blog/building-saas-applications-planetscale-netlify 2023-03-30T13:00:00.000Z 2023-03-30T13:00:00.000Z Liz van Dijk How to read MySQL EXPLAINs https://planetscale.com/blog/how-read-mysql-explains 2023-03-29T09:00:00.000Z 2023-03-29T09:00:00.000Z Savannah Longoria (query fragment): An index lookup would happen if the query had been properly parsed. (condition, expr1, expr2): An if condition is occurring in this specific part of the query. (query fragment): An index lookup would be happening via primary key. : An internal table would be created here for saving temporary results — for example, in subqueries prior to joins. MySQL EXPLAIN join types The MySQL manual says this column shows the “join type”, which explains how tables are joined, but it’s really more accurate to say the "access type". In other words, this “type” column lets us know how MySQL has decided to find rows in the table. Below are the most important access methods, from best to worst, in terms of performance: Type value Definition 🟢 NULL This access method means MySQL can resolve the query during the optimization phase and will not even access the table or index during the execution stage. 🟢 system The table is empty or has one row. 🟢 const The value of the column can be treated as a constant (there is one row matching the query) Note: Primary Key Lookup, Unique Index Lookup 🟢 eq_ref The index is clustered and is being used by the operation (either the index is a PRIMARY KEY or UNIQUE INDEX with all key columns defined as NOT NULL) 🟢 ref The indexed column was accessed using an equality operator Note: The ref_or_null access type is a variation on ref. It means MySQL must do a second lookup to find NULL entries after doing the initial lookup. 🟡 fulltext Operation (JOIN) is using the table’s fulltext index 🟡 index The entire index is scanned to find a match for the query Note: The main advantage is that this avoids sorting. The biggest disadvantage is the cost of reading an entire table in index order. This usually means accessing the rows in random order, which is very expensive. 🟡 range A range scan is a limited index scan. It begins at some point in the index and returns rows that match a range of values. Note: This is better than a full index scan because it doesn’t go through the entire index 🔴 all MySQL scans the entire table to satisfy the query Green indicates better performance, yellow indicates okay performance, and red indicates bad performance. There are also a few other types that you might want to be aware of: index_merge: This join type indicates that the Index Merge optimization is used. In this case, the key column in the output row contains a list of indexes used. It indicates a query can make limited use of multiple indexes on a single table. unique_subquery: This type replaces eq_ref for some IN subqueries of the following form:value IN (SELECT primary_key FROM single_table WHERE some_expr) index_subquery: This join type is similar to unique_subquery. It replaces IN subqueries, but it works for nonunique indexes in subqueries. The EXTRA column in MySQL EXPLAIN The EXTRA column in a MySQL EXPLAIN output contains extra information that doesn’t fit into other columns. The most important values you might frequently run into are as follows: EXTRA column value Definition Using index Indicates that MySQL will use a covering index to avoid accessing the table. Using where The MySQL server will post-filter rows after the storage engine retrieves them. Using temporary MySQL will use a temporary table while sorting the query’s result Using filesort MySQL will use an external sort to order the results, instead of reading the rows from the table in index order. MySQL has two filesort algorithms. Either type can be done in memory or on disk. EXPLAIN doesn’t tell you which type of filesort MySQL will use, and it doesn’t tell you whether the sort will be done in memory or on disk. “Range checked for each record” (index map:N). This value means there’s no good index, and the indexes will be reevaluated for each row in a join. N is a bitmap of the indexes shown in possible_keys and is redundant. Using index condition Tables are read by accessing index tuples and testing them first to determine whether to read full table rows. Backward index scan MySQL uses a descending index to complete the query const row not found The queried table was empty Distinct MySQL is scouring the database for any distinct values that might appear in the column No tables used The query has no FROM clause Using index for group-by MySQL was able to use a certain index to optimize GROUP BY operations Hands-on example of how to use MySQL EXPLAIN In this section, we will explore one way you can utilize MySQL EXPLAIN for query optimizations. To start, I created a database in PlanetScale and seeded it using the MySQL Employees Sample Database. PlanetScale is a hosted MySQL database platform that makes it easy to spin up a database, connect to your application, and get running quickly. With PlanetScale, you can create branches to test schema changes before deploying to production. This development environment, paired with some of our other tools, like Insights for query monitoring, gives you a great way to test and debug queries, leading to better performance and faster application. Sign up for a PlanetScale account. Confirm that the database is created and seeded Now that we have our database let’s run some queries. First, we’ll want to confirm that our tables are in PlanetScale. We can do this by running SHOW TABLES; in the PlanetScale CLI or web UI. For this example, I will be utilizing our web UI. Run the initial query Using a multi-column index coupled with MySQL EXPLAIN, we will provide a way to store values for multiple columns in a single index, allowing the database engine to more quickly and efficiently execute queries using the set of columns together. Queries that are great candidates for performance optimization often use multiple conditions in the WHERE filtering clause. An example of this kind of query is asking the database to find a person by both their first and last name: SELECT * FROM employees WHERE last_name = 'Puppo' AND first_name = 'Kendra'; Okay, so we know that this result isn’t ideal because it’s scanning 299,202 rows to complete the request, as shown under rows in the screenshot above. How do we go about optimizing it? We have a few different routes we can take, but only one is ideal for cost and performance. Optimization approach 1: Create two individual indexes For our first approach, let's create two individual indexes — one on the last_name column and another on the first_name column. This may seem like an ideal route at first, but there's a problem. If you create two separate indexes in this way, MySQL knows how to find all employees named Puppo. It also knows how to find all employees named Kendra. However, it doesn't know how to find people named Kendra Puppo. Some other things to keep in mind: MySQL has choices available when dealing with multiple disjointed indexes and a query asking for more than one filtering condition. MySQL supports Index Merge optimizations to use multiple indexes jointly when running a query. However, this limitation is a good rule of thumb when building indexes. MySQL may decide not to use multiple indexes; even if it does, in many scenarios, they won’t serve the purpose as well as a dedicated index. Optimization approach 2: Use a multi-column index Because of the issues with the first approach, we know we need to find a way to use indexes that consider many columns in this second approach. We can do this with a multi-column index. You can imagine this as a phone book placed inside another. First, you look up the last name Puppo, leading you to the second catalog for all the people named Kendra, organized alphabetically by first names, which you can use to find Kendra quickly. In MySQL, to create a multi-column index for last names and first names in the employees table, execute the following:CREATE INDEX fullnames ON employees(last_name, first_name); Now that we have successfully created an index, we will issue the SELECT query to find rows with the first name matching Kendra and the last name matching Puppo. The result is a single row with an employee named Kendra Puppo. Now, use the EXPLAIN query to check whether the index was used: These results show that the index was used, and only one row was accessed to fulfill this request. This is much better than the 299,202 rows we needed to access before the index. Conclusion The EXPLAIN statement in MySQL can be used to obtain information about query execution. It is valuable when designing schemas or indexes and ensuring that our database can use the features provided by MySQL to the greatest extent possible. In PlanetScale, our Insights feature + EXPLAIN statement in MySQL can be of massive assistance when you need to optimize the performance of your queries.]]> Connection pooling in Vitess https://planetscale.com/blog/connection-pooling 2023-03-27T09:00:00.000Z 2023-03-27T09:00:00.000Z Harshit Gangal How to Upgrade from MySQL 5.7 to 8.0 https://planetscale.com/blog/upgrading-to-mysql-8 2023-03-24T09:00:00.000Z 2023-03-24T09:00:00.000Z JD Lien Zero downtime Rails migrations with the PlanetScale Rails gem https://planetscale.com/blog/zero-downtime-rails-migrations-planetscale-rails-gem 2023-03-20T17:30:00.000Z 2023-03-20T17:30:00.000Z Mike Coutermarsh Preparing for MySQL 5.7 EOL https://planetscale.com/blog/preparing-for-mysql-5-7-eol 2023-03-14T13:00:00.000Z 2023-03-14T13:00:00.000Z Savannah Longoria DevOps with PlanetScale https://planetscale.com/blog/the-eight-phases-of-devops 2023-03-13T00:00:00.000Z 2023-03-13T00:00:00.000Z Brian Morrison II to discard the branch and the data it holds. The PlanetScale API offers the /organizations//databases//branches/ to delete branches if you want to use curl or other tools that can send HTTP requests. In this guide, we discussed the Test phase and dedicated database branches can be used to assist with integration testing. Next up is the Release phase, where the new software is set up for a successful deployment into production. Release phase The Release phase is where the 'Ops' part of DevOps starts. In the release phase, the primary goal is to ensure that the infrastructure and environment are set up for a successful launch of the updated software. This can include spinning up or down servers, updating operating system configurations, or setting up any other necessary infrastructure to support the application. At this point, the updated code should have been thoroughly tested and confirmed to be working to the best of the team's ability. Deploy requests in PlanetScale allow you to safely merge schema changes from a development branch into a production branch with safe migrations enabled. This allows for zero-downtime deployments of a new version of the database's schema. This is the phase where deploy requests will be utilized to prepare for a successful deployment. Deploy requests Deploy requests are used to merge schema changes from one branch to another, similar to how pull requests merge git branches. Deploy requests work by creating shadow tables that store the new version of the schema for that specific table and replicating data from the old version to the new one. This includes any writes that may occur during the process. When the data in both tables are synced up, you are presented with the option to cut over to the new version of the table, provided the auto-apply option wasn't enabled. During this phase, a deploy request should be opened from the dev branch into the production branch of your database but not applied until the Deploy phase coming up next. This will allow PlanetScale to stage the changes that need to be applied to your production database branch without affecting the current production version of your application. This means that as long as the deploy request is not "applied" to the target branch, PlanetScale will continuously keep the live table (old schema) and shadow table (new schema) in sync until your team is ready to deploy the new artifacts into production. A note on blue/green deployments If you are at the Scalar tier and above, PlanetScale databases support multiple production branches. Production branches automatically have an additional failover instance of your database ready behind the scenes to improve redundancy. While there are no tools directly within PlanetScale to assist with blue/green deployments for your database, multiple production branches can significantly reduce the administrative overhead of managing multiple MySQL environments. In this guide, we discussed the Release phase and how your production database can be set up for a live cutover using deploy requests. Next up is the Deploy phase, where all of the work is deployed to your production environment. Deploy phase Everything in the previous phases has been building to this point. It's where all the hard work gets deployed to production for the world to use. The operations team will coordinate to copy release artifacts that have been built & tested to production servers. If your team follows a blue/green strategy, the load balancers will instead start redirecting traffic to the staging server, and the current production environment takes the responsibility for staging the next cycle of application updates. Branching and deploy requests in PlanetScale are the primary features that enable PlanetScale databases for flexibility in a DevOps environment. During this phase, any open deploy requests should be closed and applied to the production branch. All of the schema changes that were configured during development should now be in production, along with the new code that requires the changes to those tables. If your organization has opted into the schema revert feature, this starts the 30-minute window where you have the option to revert the changes in case something goes catastrophically wrong. Back out of changes with schema revert Having a great deployment strategy is key to successfully implementing DevOps, but knowing how to properly back out of changes can be just as important. Many source control management systems can be automatically configured to retain a certain number of previous releases which can be used to roll back application code, but doing so for a database can be difficult without affecting the data it holds. As stated previously in this series, Deploy request utilizes shadow tables to synchronize data changes between tables with the old version of the schema and tables with the new version of the schema. If schema revert is enabled, we will continue to synchronize changes between the live and shadow tables for a period AFTER the deploy request is closed and changes applied. This enables you to quickly revert the changes made by a deploy request and instantly bring back the old version of your schema. Having this capability can significantly decrease the time to revert changes, as well as reduce the potential for your application to stay in a bad state long term. This alone can increase developer confidence when it comes to applying changes to the database. In this guide, we discussed the Deploy phase and how any open deploy requests should be closed at this point, as well as how schema revert can help back out of bad changes quickly. The next step in the DevOps cycle is the Operate phase, where the Operations team maintains the infrastructure that powers the application. Operate phase Now that everything is deployed into production and confirmed working, the operations team's main focus is keeping everything online. Ideally, this is done with a system that will monitor application load to detect spikes in usage and automatically scale resources up to keep up with the traffic. This can be accomplished with platforms like EC2 in AWS, but also on-premise with Kubernetes. All PlanetScale tiers eliminate the need to maintain your MySQL infrastructure by allowing us to do it for you. Additionally, any production branches automatically have failover replicas so even if something fails with one instance internally, a backup is always available to take over while any necessary maintenance is performed by our teams. On top of reducing the necessity of maintaining a MySQL environment, PlanetScale offers additional features that can simplify the jobs of the operations teams. Backup and restore Any well-run operations teams know that backing up and restoring data is a critical task that must be taken seriously. A time will inevitably come when data is lost whether that is due to bad code or mistakes during the deployment process. Having a way to retain snapshots of your data at specific points in time for recovery is critical, and this functionality is built into PlanetScale databases. All databases on our platform have a daily backup configured automatically, regardless of which tier you are on. Additional backups and retention periods can also be configured, with the only additional cost being the storage used by the backups. One thing that can be overlooked is the fact that backups are pointless if the data within them doesn't restore properly. Since we support the concept of database branches, and those branches are isolated instances of MySQL, restoring a backup will create a dedicated branch for the data to reside. This can vastly simplify the process of performing test restores. If you can quickly configure new environments using Infrastructure as Code tools, you can easily spin up entire production-like environments to fully test your application, which can dramatically improve the confidence of the operations team. Horizontal scaling PlanetScale is built in Vitess, which is an open-source project that enables horizontal scaling for MySQL databases. Sharding, available on our Base plan, further reducing the load on individual nodes as well as increasing performance and resiliency. Read-only regions When creating databases or branches, you'll be presented with the option to select which region you'd like your database created in. After creation, you'll also have the option to create read-only regions. This adds a replica of your database in a specific geographical location to more quickly serve queries by users in that area. Traditionally this would require operations teams to set up additional data centers linked by VPN tunnels or private ISP networks to securely synchronize data, but this is all handled by PlanetScale without such complexity. In this guide, we discussed the Operate phase and discussed features that PlanetScale offers to make the lives of Ops members easier. The last step in the DevOps cycle is the Monitor phase, where feedback and metrics are gathered for decision-making before the next iteration. Monitor phase The last phase of the DevOps cycle is to monitor the entire application. This can be by gathering feedback from customers that use the application, but also to monitor performance metrics that the application tracks. This feedback should be used in decision-making when the team inevitably comes together again to plan the next cycle. One important metric of your application's performance is how quickly your queries are executed. Slow-running queries can bring an application to its knees. PlanetScale Insights PlanetScale offers Insights with every database that is hosted on our platform, which is a visual way to see how well your queries are performing. Performance data is automatically tracked in real-time and displayed on a graph so you can see periods of high usage. You can also see which queries are executed most frequently or are taking the longest to return data. If your database is enrolled in the schema-revert feature, the metrics gathered by Insights could help in making a data-driven decision on if the schema you just deployed to production is experiencing issues and needs to be rolled back. While having your own logging and monitoring platform to analyze errors in your code is definitely a best-practice, this would act as an additional layer of analytics and may help in reducing downtime overall. PlanetScale Connect PlanetScale Connect is a feature provided to our databases that allows you to extract data from the database and safely load it into remote destinations for analytics or ETL purposes. Using Connect with our supported destinations can enable you to further process the data in any way your organization may need. This can help provide detail as to how users are using your application based on the data that's written to your database and assist in driving decisions in the next planning cycle. We currently support loading data into Airbyte and Stitch destinations, with more planned for the future. Datadog integration If you are a Datadog customer and use their platform to centralize your analytical data, we offer an integration with the service. Our integration will gather similar data that is displayed in Insights and forward it to a PlanetScale dashboard that is automatically created when the integration setup is complete. Refer to the Datadog integration article for more details on how this can be configured for your PlanetScale database. While the Monitor phase is what concludes the typical DevOps cycle, it loops back into the Plan phases for the next iteration to be set up. At this point, you should be well-equipped to make intelligent decisions on how to integrate PlanetScale into your existing pipelines, or understand how to get started with DevOps altogether! Feel free to explore more of our documentation to further your understanding of the platform. Real-world scenario DevOps is very much a "choose your own adventure" set of guidelines and that can make it confusing for teams to properly implement it given the number of choices available from code language, tooling, process, etc. The following section describes a fictitious team as they implement a new feature in their codebase. Throughout the section, we'll call out specific tools that are common in the industry to implement much of the process described in the above sections. As expected, the various features available by PlanetScale will also be described as the story progresses. This section is about a fictitious company that uses a PlanetScale database to back its application and utilizes many of the techniques discussed in the phase-specific articles in our documentation. Background The story follows Mechanica Logistics, a small warehousing and transportation company with a web application that their customers can use to place new shipping orders or track the status of existing orders. Since they are a small business, its tech team has a size to match. Jenny is their Architect and Lead Backend Developer. She primarily works with the other backend developer, Ricardo, when working on their API written with Go. Malik is the team’s designer and front-end developer and he is responsible for maintaining the React web application used by customers. Finally, Ainsley is the company’s sole Systems Engineer, responsible for maintaining the AWS infrastructure performing well. Mechanica uses the following tools in its tech team: Tool Use case Jira Organize and assign work, and create development iterations. GitHub Source control management. Slack Team messaging and system notifications. Jenkins Builds, tests, and deploys the application updates. Datadog Provides a dashboard to monitor application and infrastructure performance. PlanetScale Hosts their MySQL databases. Terraform Automate AWS infrastructure management. Atlas CLI Perform schema migrations. Infrastructure Mechanica uses AWS as its primary cloud provider, with the exception of using PlanetScale for its MySQL database. Their React front end operates as a single-page application and is stored in a dedicated S3 bucket. A CloudFront instance is used in front of it to use a custom domain name, as well as cache the front end as close to end users as possible. The API is written in Go and is running on two Linux EC2 instances in production. A technique called “blue/green” is used with the API, so one instance is always live and the other is used as the staging server. There are three environments active at any time. A development environment is used for building and testing new functionality by the developers. A test environment is used by Jenkins to run automated tests to ensure that everything is built according to spec. Finally, there is the production environment that's used by Mechanica customers. Although there are three separate environments, a single PlanetScale database is used, with a separate database branch configured for each environment. The production database branch also has safe migrations enabled. This prevents accidental changes to the schema by enforcing the use of the PlanetScale flow, requiring that schema changes be made using branching and deploy requests. The request One of Mechanica’s biggest partners, Empress Products, recently experienced large unexpected growth and their shipping orders likewise increased. Due to the increase in orders, the systems at Empress were struggling to continuously poll for order status using Mechanica’s API and needed another solution. The tech team at Empress submitted a request that Mechanica figures out a way to send them updates on order status whenever things change instead. Since Empress was one of their largest customers, they decided to prioritize it and address it during the next development cycle. Plan and Code Early Monday morning, the team at Mechanica assembled as they do every two weeks to decide what needed to get done in this development cycle. Jada, the company project manager, was also present as usual to provide insight on the feedback they’ve gathered from Mechanica customers. Jada informed the team of the request from Empress. After some brainstorming among the technical team, they settled on building a system that used webhooks, a way to allow the systems at Mechanica to submit status updates to any HTTP endpoint at the point when an order changes, in near real-time. As the planning concluded around the new system, the team identified the following required changes: Update the front end to allow customers to register webhooks. Update the Customers table in the database to add columns for storing the webhook endpoints and signing keys for the webhooks system. Create a new serverless function to process outgoing webhook messages, signing the messages and sending them to the customer endpoint. Add a message queue to offload messages to buffer messages between the API and serverless function to reduce API load. Identify anywhere in the current API that order statuses change to submit a message to the message queue. As soon as the Sprint was created and confirmed, a Jira automation would use the PlanetScale API to create a fresh dev database branch for the team to begin working with. The most recent backup would also be specified to seed data into that branch, giving the team an isolated environment that mirrored production. Each member of the team was assigned work relevant to their expertise. Malik built the necessary views required for the React application. This included views to create webhook endpoints, manage and delete existing endpoints, and generate signing keys as needed. Jenny and Ricardo worked on building the backend components. The new serverless function would be written in Go and would be responsible for using the signing key to sign webhook messages and POST them to customer endpoints. The two were also able to identify where changes in the existing API code were needed to allow the API to dispatch messages into the message queue. Ainsley takes the security of Mechanica databases very seriously and they do not give out connection strings, even to developers in case one of their systems gets infected. Due to this policy, Jenny and Ricardo proxy connections to the PlanetScale database using the pscale connect command of the CLI. When Jenny and Ricardo are working on the backend services, they simply run pscale connect to set up a tunnel to the database before they start their local development instance of the APIs. This allows the locally running instances of their APIs to connect to localhost, where the PlanetScale CLI will redirect the queries directly into the database without having to use connection strings. The backend team also updates the schema definition file to add the new table that was required and used the Atlas CLI to apply the database changes to the dev branch of their database. This will ensure that the state of the database is always consistent and reviewable by the team (since the definition is managed by source control) instead of having developers apply changes manually and make mistakes. Ainsley worked to build out a Terraform definition that would be used to not only create the new infrastructure components in AWS but maintain them going forward so that they didn't have to manually tweak settings as things changed over time. Along with Jenny’s help, the two of them were able to quickly update the configuration file for the API to add credentials allowing the API to submit messages to the queue, as well as deploy the new serverless function into the development environment for some live testing by the developers. Once everything was built and manually tested by the developers, it was time to open a pull request for the monorepo and review all of the changes as a team. Since the team had been working together for several years at this point, only minimal changes needed to be made before the pull request was closed and it could move into testing. Build and Test At the moment the pull request closed, GitHub used a webhook to notify the Jenkins server to build the newest version of the code. Jenkins then cloned down the repo from GitHub at that specific commit where the PR was merged and compiled the API project and the new serverless function into their respective binaries and uploaded the artifacts to a dedicated AWS S3 bucket to store for usage throughout the pipeline. Once the build stage of the pipeline was completed, it was time to move on to testing. Ainsley had previously spent weeks ensuring that the entire testing process was also automated by Jenkins. Since the team had taken a test-driven development approach to build the code, it had plenty of unit and integration tests built to ensure that the new code met the business requirements set during planning. The process kicked off by running a Terraform command that would spin up the necessary infrastructure in AWS for testing. This would create an SQS queue in a dedicated AWS test account that could be used during integration testing to make sure the webhooks feature was built to spec. Next up would be building out the test database infrastructure. Using the PlanetScale CLI, Jenkins would create a replica of the main production database branch by creating a new branch called test. This would automatically create an isolated MySQL environment where integration testing could be performed without affecting production. In the past, the team used to have a .sql script that would seed test data to their test branch for running this process, but more recently they’ve been using the Data Branching® feature set to restore the most recent backup of the main branch into test, creating an identical copy of their production database. To finalize the setup of the database, Jenkins would run the Atlas CLI to sync up the new table from the dev branch into test. Now the database looks exactly as it would once all of these changes make it into production. Before running the test, the proper credentials needed to be generated and added to the project configuration. Jenkins would again use the PlanetScale CLI to generate a connection string and store it alongside the project. Next, Jenkins would use the cloned repository and run the go test command to execute all of the tests the team had written. This would not only be the unit tests that would validate business logic, but also the integration tests that would perform CRUD logic for storing and reading webhook configuration from the database, as well as simulating an order to check that the message gets processed as expected. Once the tests have concluded and all have passed, Jenkins would use Terraform to tear down the test infrastructure in AWS and the PlanetScale CLI to delete the test branch since it is no longer required. Finally, Jenkins would once again use the PlanetScale CLI to open a Deploy request from dev into main, then notify the team using Slack so they could prepare for deploying the latest version of their application to production. Release and Deploy After Jenny, Ricardo, Malik, and Ainsley reviewed the test results and confirm everything went smoothly, they approve the Deploy request so PlanetScale can start synchronizing the changes from the development environment into production. Since this process uses shadow tables to effectively stage changes without making them live, the actual process of going live happens quickly and painlessly. At this point, the latest version of the code has been thoroughly tested and the schema changes have been staged for the database. Ainsley logs into Jenkins and approves the final phase of the pipeline to deploy all of the changes to production. This kicks off a process where Jenkins utilizes deploy agents installed on the production EC2 servers to download the latest artifacts from S3, replace the old binaries, and restart the service that keeps the API alive. The script also creates the necessary SQS queue in AWS using Terraform and uses the PlanetScale CLI to apply the schema changes from the deploy request, which effectively cuts over the application to use the new version of the schema. Finally, the load balancer is updated to reroute traffic to the newest version of the application. After a week and a half of hard work, the changes are now live and can be used by Mechanica customers. Operate and Monitor Although the code has already gone through a rigorous testing process, it's inevitable that certain issues can occur once the application hits production as there are certain variables that simply can't be accounted for in testing. Upon deployment, Ainsley starts to monitor the Datadog dashboard configured to store the logs forwarded from AWS as well as Insights data forwarded from PlanetScale. This was important since the window to revert schema changes is open for 30 minutes, allowing for quickly rolling back changes. The dashboard includes metrics detailing the operating capacity of the EC2 servers, network traffic, application errors, and query performance metrics. Since moving to PlanetScale, the team hasn’t had much to worry about regarding the database infrastructure since that is completely managed for them. This has freed much of Ainsley’s time to focus on optimizing the performance of other infrastructure components, so nearly all issues have been ironed out. As the new feature started to be utilized, Ainsley did notice that some queries weren’t performing as expected based on analytical data being forwarded to Datadog from PlanetScale. Ainsley opened the Insights tab of the database to validate the data in their dashboard and indeed notice that the query for webhook configurations was performing a scan on the entire table instead of just the necessary rows. They decided to add a new issue to the Jira board to address it in the next cycle. Although there was minor room to improve, the feedback from Empress Products on the new feature was overwhelmingly positive, and that they wanted this same functionality built into many other areas of the application. Jada took the feedback and added yet another issue in Jira to make in the future. Conclusion Although this story is fictional, it demonstrates how DevOps and PlanetScale can help streamline team processes and ease the pain of deploying applications into production. After reading this, you should have a better understanding of how these practices can be used within your organization.]]> Using MySQL with SQLAlchemy: Hands-on Examples https://planetscale.com/blog/using-mysql-with-sql-alchemy-hands-on-examples 2023-03-07T14:00:00.000Z 2023-03-07T14:00:00.000Z Anthony Herbert ://:@:/dbname Install the Python MySQL database driver Since SQLAlchemy works with many different database types, you'll need an underlying library, called a database driver, to connect to your database and communicate with it. You don't have to use this driver directly, because as long as SQLAlchemy has the correct driver, it will automatically use it for everything. The Python MySQL Connector is used as the driver in this tutorial, but other good ones are PyMySQL and MySQLdb. You'll need to install your driver with pip.pip install mysql-connector-python So let's say your username, password, hostname, port, and database name are user1, pscale_pw_abc123, us-east.connect.psdb.cloud, and 3306, respectively. Your connection string would look like the following if you were using mysqlconnector as your driver to connect to a database named sqlalchemy.mysql+mysqlconnector://user1:pscale_pw_abc123@us-east.connect.psdb.cloud:3306/sqlalchemy Create SQLAlchemy engine object Once you have your driver installed and your connection string ready to go, you can create an engine like this:from sqlalchemy import create_engine connection_string = "mysql+mysqlconnector://user1:pscale_pw_abc123@us-east.connect.psdb.cloud:3306/sqlalchemy" engine = create_engine(connection_string, echo=True) Typically, you don't need echo set to True, but it's here so you can see the SQL statements that SQLAlchemy sends to your database. By default, SSL/TLS usage in mysql-connector-python is enabled, which is required to connect to PlanetScale. This means you do not need to pass it into create_engine() as a connection arguement. See the Python connection arguments MySQL docs for more info and to see all of the possible arguments. If you run the code and get no errors, then SQLAlchemy has no trouble connecting to your database. If you get an error like "access denied" or "server not found," then you'll need to fix your connection string before proceeding. With the engine object working, you can then continue with SQLAlchemy in various ways. This article covers how to use it to send raw queries to your database and how to use it as an ORM. Raw SQL statements in SQLAlchemy Now that we have our engine object working let's use it to send raw SQL statements to the database and receive the results in return. Create a connection object To start sending queries over, you'll need to create a connection object. Since the engine manages connections, you need to ask the engine for a connection before you can send statements over to the database. We need to call engine.connect() to get a connection, and because the connect method is defined with a context manager, we can use a with statement to work with the connection.with engine.connect() as connection: Now that you have the connection, you can execute any SQL statement that works on your database by importing the text function from SQLAlchemy. Add from sqlalchemy import text in your Python file. You then pass the query as a string to the text function and then finally pass the text function to connection.execute(). Create a table To create a table, you can run the following code:connection.execute(text("CREATE TABLE example (id INTEGER, name VARCHAR(20))")) If you run that and get no errors, that means the table was created. If you try to run the code again, you'll get an error saying the table already exists. You can also go back to your PlanetScale dashboard to confirm the table was added. Click on your database, click "Console", connect to your branch, and run the following:SHOW tables; Add data to a table Next, let's insert some data into our new table.connection.execute(text("INSERT INTO example (name) VALUES (:name)"), {"name": "Ashley"}) connection.execute(text("INSERT INTO example (name) VALUES (:name)"), [{"name": "Barry"}, {"name": "Christina"}]) connection.commit() The first execute statement will create just one row because a single dictionary was passed. But the second statement will create two rows since a list of two dictionaries was passed. Just make sure the keys in the dictionary match the placeholders you have with a colon in front of the name. Even though we aren't dealing with user input here, it's still a good idea to make a habit of passing in parameters instead of the data directly inside of an insert statement or select statement. Unlike CREATE statements, INSERT statements happen in transactions, so we have to save them to the database by calling .commit() after we execute our insert statements. Query data Finally, now that we have some data in the database, let's go ahead and query that data so we can see it again. We can assign the result of connection.execute() to a variable called result, and if we loop over result.mappings(), we'll see that we get dictionaries for each row, where the key in each dictionary represents the column name. This makes it easy for us to retrieve the data and display it in a loop.result = connection.execute(text("SELECT * FROM example WHERE name = :name"), dict(name="Ashley")) for row in result.mappings(): print("Author:" , row["name"]) As you can see, you only need to know a few things to write raw queries using SQLAlchemy. If you want to use it as an ORM, you can do that as well. Using SQLAlchemy ORM to write queries The idea behind ORM (object-relational mapping) is to create a code representation of your database using classes and objects instead of writing raw SQL statements. The classes represent the tables in your database, and the objects of those classes represent rows. So the first step to using ORM is to define classes that map to your tables. Classes that represent tables in an ORM are called models. Before we can do the mapping, we need something called the DeclarativeBase from SQLAlchemy. Even though our classes could inherit directly from DeclarativeBase, we will instead create our own Base class that inherits it and then call pass. This makes it easy to add additional settings to our Base class in the future since all of our models will inherit from this one.class Base(DeclarativeBase): pass Create models Now we can create our models. Let's first create an Author model which will map to an author table. The idea here is to first define a tablename, which is the attribute __tablename__. Define the columns Next, we need to define the columns. Starting in SQLAlchemy 2.0, we can use the Python typing system to define the columns for us. So the format of each column is the name of the column, followed by a class called Mapped with the Python type that closest matches the type you want in the database. So for an ID column, we would have id: Mapped[int]. Next, that attribute is going to be set equal to the mapped_column function call, where we could set additional properties on our column like primary key, max length, nullable, etc. So let's create an Author model with two fields: id and name, which means we'll have a table with two columns. SQLAlchemy requires each model to have a primary key so it can internally keep track of each object, so let's make ID the primary key.class Author(Base): __tablename__ = "author" id: Mapped[int] = mapped_column(primary_key=True) name: Mapped[str] = mapped_column(String(30)) Handling relationships and foreign key constraints PlanetScale now supports foreign key constraints. This information below is out of date, but it will still correctly work. Next, let's create a Post model, which will have a relationship to the Author model. We can create an author_id column inside of Post that holds the reference to the author who created the post. For most database systems, you'd pass ForeignKey to mapped_column to create an actual foreign key constraint in the database. But with PlanetScale, we don't recommend using foreign key constraints. However, we can still use SQLAlchemy to manage the relationship for us. Since databases with foreign key constraints are very common, variations for those databases are included in the commented-out lines.class Post(Base): __tablename__ = "post" id: Mapped[int] = mapped_column(primary_key=True) title: Mapped[str] = mapped_column(String(30)) #author_id: Mapped[int] = mapped_column(ForeignKey("author.id")) author_id: Mapped[int] The advantages of having no foreign keys are we can have multiple versions of our database schema in the same way we have multiple versions of code through things like git branches. It also allows us to make schema changes to production databases without any downtime. And finally, it makes it easier to scale the database through sharding. But even without a foreign key, we can still have a relationship between two tables. To get SQLAlchemy to manage the relationship for us, we can create a relationship attribute. Unlike the attributes for the columns, no column gets created in the database. Instead, the relationship only exists in our code while it's running. So we can add a relationship attribute and the type will be a list of Author classes, which we can pass to the Mapped class as the type.posts: Mapped[list["Post"]] = relationship(primaryjoin='foreign(Post.author_id) == Author.id') Since we're not using ForeignKey, we need to tell SQLAlchemy how to handle our relationship. We can do that we the primaryjoin argument to relationship. If we used a database with foreign keys, then the ForeignKey being passed to the mapped_column would be enough. We can also create a Tag model in a similar way. This Tag model represents tags that each post could have.class Tag(Base): __tablename__ = "tag" id: Mapped[int] = mapped_column(primary_key=True) text: Mapped[str] = mapped_column(String(30)) Because one post can have many tags and one tag can belong to many posts, we need to create a many-to-many relationship. We can create a post_tag table to represent this relationship. Many-to-many relationships have the foreign key stored in a separate table called an association table. We can create that table directly in SQLAlchemy. You could also use a model for this, but it's better to use a table because you won't be working with this table directly. Instead, SQLAlchemy will automatically manage the data in this table for you by using the relationships you define. You can create the table like this:post_tag = Table( "post_tag", Base.metadata, #Column("author_id", ForeignKey("author.id"), primary_key=True), #Column("tag_id", ForeignKey("tag.id"), primary_key=True), Column("post_id", Integer, primary_key=True), Column("tag_id", Integer, primary_key=True) ) Create tables with create_all() Now that we have the Tag table defined, to create the tables in the database, you can call create_all on your Base class. The create_all call takes an engine object, so you can reuse the one we created earlier. Create_all will take about DeclarativeBase and instruct it to create statements for each one of our tables and add them to the database. You'll see that printed to your terminal when you run.Base.metadata.create_all(engine) With our tables created, we can go ahead and insert data into the tables and then query the tables. Since we're working with the ORM, the way to create new rows is first by creating objects. So for example, to create a new Author, we can instantiate an author object. For relationships, we can set one object to be related to another when we instantiate the related object. We want to use the relationship attribute instead of the _id field directly because SQLAlchemy will take care of the ID field for us. For many-to-many relationships, we append to the relationship attribute like it's a Python list. We only need to append children of the relationship. To add them to the database, we need to first add them to the session. Finally, we need to call commit to save them to the database. When you run the script, you'll see the actual insert statements being printed.with Session(engine) as session: author = Author(name="David") post = Post(title="Python Essentials", author=author) session.add(author) session.add(post) post2 = Post(title="SQL Secrets", author=author) post3 = Post(title="Advanced MySQL", author=author) session.add_all([post2, post3]) tag1 = Tag(text="python") tag2 = Tag(text="sql") tag3 = Tag(text="mysql") session.add_all([tag1, tag2, tag3]) post.tags.append(tag1) post2.tags.append(tag2) post3.tags.append(tag2) post3.tags.append(tag3) session.commit() Query the data Now that we have some data in the database, we can go ahead and query that data. First, we write our query statement. Start by passing your Model call to the select function. Then you have the option to use the where attribute on the resulting object. We can then pass all of that to session.scalar to run the query. With the result of scalar, we can print out the results to the terminal. We can also look at the values in the relationship. For each post, we can look at the tags as well. If you want to leave out the where and get all of the posts, you will use scalars instead of scalar. Then we can loop over the object returned and print out all the titles.stmt = select(Author).where(Author.name == "David") author = session.scalar(stmt) for post in author.posts: print(post.title) for tag in post.tags: print(" ", tag.text) for post in session.scalars(select(Post)): print(post.title) Conclusion With your new knowledge of SQLAlchemy, you should have a good starting point to continue using it in any Python project that uses a SQL database. As long as you can set up the engine object, you'll be able to decide whether you want to simply send raw SQL statements, construct SQL statements using the SQLAlchemy API, or map your Python classes and objects to your database tables and data.]]> Improvements to database branch pages https://planetscale.com/blog/improvements-to-database-branch-pages 2023-03-01T14:23:00.000Z 2023-03-01T14:23:00.000Z Jason Long Announcing Vitess 16 https://planetscale.com/blog/announcing-vitess-16 2023-02-28T15:50:00.000Z 2023-02-28T15:50:00.000Z Vitess Engineering Team What are the disadvantages of database indexes? https://planetscale.com/blog/what-are-the-disadvantages-of-database-indexes 2023-02-17T14:45:00.000Z 2023-02-17T14:45:00.000Z JD Lien Faster MySQL with HTTP/3 https://planetscale.com/blog/faster-mysql-with-http3-video 2023-02-16T13:00:00.000Z 2023-02-16T13:00:00.000Z Matt Robenolt Migrating from Postgres to MySQL https://planetscale.com/blog/migrating-from-postgres-to-mysql 2023-02-09T15:29:55.100Z 2023-02-09T15:29:55.100Z Adnan Kukic Introducing the PlanetScale API and OAuth applications https://planetscale.com/blog/introducing-planetscale-api-and-oauth-applications 2023-01-31T14:00:00.000Z 2023-01-31T14:00:00.000Z Frances Thai Taylor Barnett Beta features page. Once we've received your enrollment request, a PlanetScale team member will be in touch about your OAuth use case. Refer to our OAuth documentation for further instructions on creating an OAuth application and completing our authorization flow. PlanetScale API + OAuth application demo We've created a Next.js-based demo called PlanetPets that uses PlanetScale OAuth and API to access users' organizations, databases, branches, and create new branches. The user's organizations are then presented as "gardens" where their databases are "trees." Within PlanetPets, users can water their "trees" to grow new branches. This sample app shows you how to implement OAuth authentication with PlanetScale in a Next.js application. Set it up yourself using the code in the PlanetPets GitHub repo, or play around with the PlanetPets live demo. PlanetScale integration examples Two fantastic community partners have already built integrations using the powerful combination of OAuth applications and the PlanetScale API. These integrations are available for use today. Netlify Netlify is launching a new PlanetScale integration into Netlify Labs. Netlify's new integration allows Netlify users to closely integrate PlanetScale branches, deploy requests, passwords, and other features into the Netlify workflow. Additional benefits for Netlify users include more easily connecting PlanetScale databases to Netlify sites, assigning database branches to different deploy contexts, and using the withPlanetScale function in Netlify Functions to seamlessly insert a connection into the database call. You can read more about the integration in the Netlify integration docs. Resmo Resmo uses an OAuth application and the PlanetScale API to connect to PlanetScale in a few clicks to bring asset visibility, continuous security, and compliance of PlanetScale databases to their users. Resmo collects directory assets like databases, organizations, and database branches from users' PlanetScale accounts through the API for users to query and set up custom security rules to automate security checks. You can read more about the integration in the Resmo integration docs. Feedback We can't wait to see what you'll build with the new PlanetScale API and OAuth applications! If you have feedback on your experience using the API, we would love to hear it. You can open up a new discussion topic in the PlanetScale discussion repo with your feedback.]]> Common MySQL errors and how to fix them https://planetscale.com/blog/common-mysql-errors-how-to-fix-them 2023-01-27T14:00:35.694Z 2023-01-27T14:00:35.694Z Mary Gathoni wait_timeout value determines how long the server waits before closing a connection due to inactivity. To fix this, check the wait_timeout value (28800 seconds by default) and increase it if it’s too low. Error 2008: Client ran out of memory This error message means there’s not enough memory to store the entire query result. To solve this problem, check the specifics of the query. Do you need to return this many results from the database? If not, modify the query to return only the necessary rows. Error 2013: Lost connection during query Error 2013 occurs when the connection drops between the MySQL client and the database server, usually because the database took too long to respond. To fix the error, first, check that your internet connection is stable. Delayed results may be due to network connectivity issues. Also, try increasing the net-read-timeout value to give more time for the query to complete. Non-coded errors In addition to coded errors, there are several common non-coded MySQL errors that you might encounter. Packet too large The maximum possible size of a packet transmitted to or from a MySQL 8.0 server or client is 1GB. The max_allowed_packet variable stores the allowable packet size. For the client, the default max_allowed_packet value is 16MB and for the server is 64MB. To fix this error, increase the max_allowed_packet value for the client and the server. For example, increase the max_allowed_packet for the client to 32MB:mysql --max_allowed_packet=32M Note that MySQL needs to restart for the change to take effect. Can’t create/write file You’ll get this error when MySQL can’t create the temporary file for the result in the temporary directory. This might be because there’s no memory left in the /tmp folder, or if there’s an incorrect configuration that doesn’t allow MySQL to write in the /tmp folder. To solve the memory issue, try starting the MySQL server with the --tmpdir option and specifying a directory for the server to write to. For example, to specify C:/temp:tmpdir=C:/temp If the configuration is incorrect, make sure MySQL has permission to write to the directory specified by tmpdir. Commands out of sync The commands out of sync error occurs when you call client functions in the wrong order. For example, using mysql_use_result() before calling mysql_free_result() will raise this error. To fix this error, check your functions and make sure you are calling them in the correct order. Hostname is blocked This error occurs when the MySQL server receives too many connections that have been interrupted by the host. The server assumes something is wrong, like someone trying to break in, and blocks the hostname until you execute the flush-hosts command. The number of interrupted connect requests is determined by the max_connect_errors variable, 10 by default. Modify the value by starting the MySQL server like this:mysqld_safe --max_connect_errors=10000 Aborted connections This error occurs when clients attempt and fail to connect to the MySQL server, often due to the client using incorrect credentials or lacking access privileges. To fix this error, start by checking the error logs and general logs at /var/log/mysql/ to determine the cause of the aborted connections. Conclusion Error handling can be exhausting and time-consuming, so it's important to understand how to fix the common MySQL errors. If you're looking for a straightforward, developer-friendly way to run MySQL, try PlanetScale. You can import your application's existing database with no downtime using our Import tool and be up and running in no time.]]> MySQL scaling made easy https://planetscale.com/blog/mysql-scaling-made-easy 2023-01-26T13:00:00.000Z 2023-01-26T13:00:00.000Z Jonah Berquist What is the N+1 Query Problem and How to Solve it? https://planetscale.com/blog/what-is-n-1-query-problem-and-how-to-solve-it 2023-01-18T08:01:46.798Z 2023-01-18T08:01:46.798Z JD Lien connect(); $sql = "SELECT * FROM categories;"; $stmt = $conn->prepare($sql); $stmt->execute(); Second query — Looping over each category and grabbing the items:fetch()) { // Show category name echo $row['name']; // Now query for the items for this category $sql = " SELECT id, name FROM items WHERE category_id = :category_id ORDER BY name; "; $stmt2 = $conn->prepare($sql); $stmt2->bindParam(':category_id', $row['id']); $stmt2->execute(); $rowCount += $stmt2->rowCount(); while ($row2 = $stmt2->fetch()) { // Show item ID and name echo $row2['id']; echo $row2['name']; } } This approach has the benefits of having two simple queries and clear, procedural code. Unfortunately, this approach is flawed, and you should avoid this situation where you are executing many database queries in a loop. What caused the N+1 query problem? This type of query execution is often called "N+1 queries" because instead of doing the work in a single query, you are running one query to get the list of categories, then another query for every N categories. Hence the term "N+1 queries". In the above example, our database contains about 800 items across 17 categories. It takes over 1 second to run the 18 simple queries involved in this! That's pretty slow. If you have more complex queries with a lot of data, it will take even longer. For this simple example, it's possible to perform the exact same job 10× faster by using only one query that uses a JOIN clause. We could refactor the above code to look something like this:connect(); // Record the time before the query is executed $timeStart = microtime(true); $sql = " SELECT c.id AS category_id, c.name AS category_name, i.id AS item_id, i.name AS item_name FROM categories c LEFT JOIN items i ON c.id = i.category_id ORDER BY c.name, i.name; "; $stmt = $conn->prepare($sql); $stmt->execute(); $rowCount = $stmt->rowCount(); $lastCategoryId = null; while ($row = $stmt->fetch()) { // Render the heading for each category if this category is new if ($row['category_id'] != $lastCategoryId) { echo $row['category_name']; } // Display the row for each item if (!is_null($row['item_id'])) { echo $row['item_id']; echo $row['item_name']; } $lastCategoryId = $row['category_id']; } With this update, we accomplished much the same work with a single, slightly more complicated query. Attempting our demo of this again, we can observe a significant performance difference between the original page and this one! The page loads in about 0.16 seconds, instead of 1.4 seconds. In this simple example, with a database that isn't very large, the n+1 approach takes about 10 times longer! Imagine you had thousands, or millions of records. The performance delta could be the difference between a reasonable load time and a page that takes so long to load that it causes a timeout on the server. Creating data structures for more complicated queries Sometimes you may have a more complicated operation in mind. Say you wanted to show the categories along with the count of each item in each category. You could use an aggregate query (GROUP BY), as shown below:SELECT c.id, c.name, count(i.id) AS item_count FROM categories c LEFT JOIN items i ON c.id = i.category_id GROUP BY c.id, c.name ORDER BY c.name; But then how would we also get the list of items from a query like this where we are grouping? While it's often most efficient to let the database server do a lot of the heavy lifting instead of our server-side code, for something like a simple count of items, it may not be necessary. If we actually just queried for the items, it's pretty easy to let the server-side code (PHP, in our example) do the count for us! We can refactor this such that we do the job with a single query, then turn that query into a clean data structure.connect(); // Record the time before the query is executed $timeStart = microtime(true); $sql = " SELECT c.id AS category_id, c.name AS category_name, i.id AS item_id, i.name AS item_name FROM categories c -- Using a normal JOIN would not get the categories with 0 items LEFT JOIN items i ON c.id = i.category_id ORDER BY c.name, i.name; "; $stmt = $conn->prepare($sql); $stmt->execute(); $rowCount = $stmt->rowCount(); $lastCategoryId = null; $lastCategoryName = null; // Build a 2D array of categories with their items $categories = []; // A categoryItems array will become the value for each category $categoryItems = []; // Alternative approach: build a data structure with the data we want as a 2D array. while ($row = $stmt->fetch()) { // Render the heading for each category if this category is new if (!is_null($lastCategoryId) && $row['category_id'] != $lastCategoryId) { $categories[$lastCategoryName] = $categoryItems; // Reset the categoryItems array $categoryItems = array(); } // Create an array of all the non-null items if (!is_null($row['item_id'])) $categoryItems[$row['item_id']] = $row['item_name']; $lastCategoryId = $row['category_id']; $lastCategoryName = $row['category_name']; } // Add the last category to the array with its items $categories[$lastCategoryName] = $categoryItems; Now that we have this $categories array with arrays of items within, we can do a nested loop to render the data in the way we see fit. When we want the count of items, you can simply run count($items) to get the quantity. $items) { echo $categoryName; // Show the count of items in the category echo count($items) . ' items'; if (count($items)) { // Loop through all the items in the category and display them foreach($items as $itemId => $itemName) { echo $itemId; echo $itemName; } } Using techniques like this, you can keep your page load times quite fast by being efficient with your use of the database. Instead of writing your code such that you have 1 query plus another for each record of that query, it is well-worth the effort to write your code such that you have 1 query that returns all the data you need. Using this approach, you can also create data structures that are more useful for your application. For example, you may want to create a data structure that is keyed by the category ID, and then have the items as sub-arrays. This would allow you to easily access the items for a specific category by its ID. Identifying N+1 queries If you have a more complex application, you may have a lot of N+1 queries and not know it. There are a few ways to identify these queries and fix them. If you're working on a Laravel app you can use Laravel Debug Bar. Laravel also allows you to fully disable N+1 queries by adding the following line to your AppServiceProvider inside the boot method:Model::preventLazyLoading(!app()->isProduction()); This will cause the application to throw an exception if it detects an N+1 query when not in production, allowing you to detect and fix these issues. PlanetScale Insights PlanetScale also offers an analytics and monitoring solution called PlanetScale Insights. This is accessible from your PlanetScale dashboard and allows you to see the queries that are being run on your database. Using this, you can identify many types of issues with your queries, including N+1 queries and long-running queries. The screenshot below is from the demo database we've been using in this article. The first query is our more complex but efficient JOIN query, which read 834 rows, returned 815 rows, and took a total of 14ms. The two queries below that are inefficient queries that resulted in the N+1 problem. Together, they took a total of 42ms and 13,889 rows read to give us the same results as the more complex query. Overall, this shows us right away that our N+1 queries: Ran way too many times Read way more rows than returned And performance was relatively slow Now you know how to identify N+1 queries, how to fix them, and how to use PlanetScale Insights to monitor your queries and identify performance issues so you can get out there and write some fast, lean code!]]> Support’s notes from the field https://planetscale.com/blog/supports-notes-from-the-field 2023-01-11T14:45:00.000Z 2023-01-11T14:45:00.000Z Mike Stojan : in use: in use: for tx killer rollback (CallerID: planetscale-admin) When you hit the 900 seconds query timeout instead, you will see an error message similar to this one:target: example-db.-.primary: vttablet: rpc error: code = Canceled desc = (errno 2013) due to context deadline exceeded, elapsed time: 15m0.002989349s, killing query ID 65535 (CallerID: ) These timeouts can be reached with complex transactions or queries, but most of the time, it's rather the user's application keeping the transaction open while handling other tasks such as data manipulation or sorting instead of closing the transaction first. Loops such as while or until , or for loops are particularly susceptible to that. There is a workaround to lift our configured timeouts and that is to change the workload mode from OLTP (Online transactional processing) to OLAP (Online analytical processing). We generally recommend against using it, though, as it can cause rather drastic side effects such as a workload consuming all available resources or blocking other important, short-lived queries or transactions from completing, or overloading a database up to a point where it goes into an unrecoverable state and where manual intervention is needed. It can also block planned failovers or critical updates and will make it easier to hit other intentional limits or timeouts dictated by MySQL. Again, we do not recommend changing your database's workload mode. There is almost always a better solution. However, if you still want to try this out, you can switch to OLAP by issuing a set workload='olap'; on a per-session basis, meaning you would have to directly execute it before running the affected transaction. The workload cannot be changed globally, and it will reset to OLTP after you have closed the session. The best long-term solution still is to optimize your database's schema and your application's transactions and control structures to make its workloads fit into the 20 seconds time window. For simple workloads, consider using optimistic locking instead of transactions, and for more complex workloads, consider adopting Sagas. For large ETL workloads, we support data integration engines such as Airbyte and Stitch, with which you can offload these processes to other platforms that are more specialized in this field. To help you with optimizing your queries and transactions, PlanetScale provides you with additional tools such as Insights. And, if you need a hand with any of this, please open a ticket with us or open an issue in our discussions board.]]> Solving N+1’s with Rails `exists?` queries https://planetscale.com/blog/rails-n1-exists 2023-01-10T17:30:00.000Z 2023-01-10T17:30:00.000Z Mike Coutermarsh { where(name: PRELOADED_FLAGS) }, as: :target, class_name: "BetaFeature" You can now replace beta_features with preloaded_beta_features to load in only the records you need.]]> Faster MySQL with HTTP/3 https://planetscale.com/blog/faster-mysql-with-http3 2023-01-04T14:45:00.000Z 2023-01-04T14:45:00.000Z Matt Robenolt What is a query planner? https://planetscale.com/blog/what-is-a-query-planner 2022-12-15T14:45:00.000Z 2022-12-15T14:45:00.000Z Andres Taylor Temporal workflows at scale: Part 2 — Sharding in production https://planetscale.com/blog/temporal-workflows-at-scale-sharding-in-production 2022-12-14T00:03:57.138Z 2022-12-14T00:03:57.138Z Savannah Longoria Rails’ safety mechanisms https://planetscale.com/blog/rails-safety-mechanisms 2022-12-12T00:43:00.000Z 2022-12-12T00:43:00.000Z Jason Charnes <%= form.text_field :name %> <%= form.text_field :location %> <%= form.submit %> <% end %> The parameters sent from the form to the controller look like this:{user: {name: "Jason Charnes", location: "Memphis"}} We can pass the user hash directly:class UsersController < ApplicationController def create User.create!(params[:user]) end end 🚨 This is mass assignment; it’s dangerous. While creating the record, we’re mass assigning the attributes. This is simple, clean, and appears to work great, but there’s a problem. The User model also has an admin boolean that defaults to false. What if our end-user is sneaky and adds this to the form? Now the params look like this:{user: {name: "Jason Charnes", location: "Memphis", admin: "true"}} We’re passing the entire list of parameters to Active Record, including admin.class UsersController < ApplicationController def create @user = User.create!(params[:user]) @user.admin? #=> true end end Yikes, we gave away the keys to the kingdom. This is the role strong parameters play. Strong parameters Instead of mass assignment (passing the raw params directly to Active Record), we use strong parameters to signal which attributes are allowed.class UsersController < ApplicationController def create User.create!(user_params) end private def user_params params.require(:user).permit(:name) end end Here, we define a user_params method that is responsible for allowing specific parameters through. We start by telling the params object that we require the parameters to have a key of :user. If the params don’t have this key, an error is raised and the request is halted. From there we provide a list of permitted attributes to the permit method. Now, if our end-user tries to pass admin as a parameter, Rails will exclude it from the list of params passed to Active Record. Rails will log unpermitted parameters by default. This is useful for debugging in development and looking for bad actors in production. If you want to take things further, you can tell Rails to raise an error if unpermitted parameters are passed:config.action_controller.action_on_unpermitted_parameters = :raise Strong parameters are a controller-level feature. Is there a way to enforce this in the model? Rails doesn’t provide this, but it used to. Before the introduction of strong parameters in Rails 4, mass assignment protection happened at the model level. (Does anyone remember attr_protected and attr_accessible?!) The strong parameters pattern is more flexible. You likely want to permit different attributes depending on the context. A user shouldn’t set their admin status. But an existing admin may be able to set it. While it’s easier to define it in the model, it’s simpler to let the controller do it. N+1 prevention in Rails N+1 queries are unfortunately easy to perform in Active Record. ✨ If we access an association we haven’t loaded, Rails will make the database lookup on our behalf. Rails has our back here. It comes at a cost, though. (Unless you view N+1s as a feature.) Pretend we’re rendering a list of orders:class OrdersController < ApplicationController def index @orders = Order.all end end <% @orders.each do |order| %> <%= order.id %> <%= order.customer.name %> <%= order.created_at %> <% end %> We’re rendering the customer’s name on each order. An order belongs to a customer.class Order < ApplicationRecord belongs_to :customer end We didn’t ask the database for customers, though. Just orders. The logs show the single database query for orders. ✅SELECT "orders".* FROM "orders" And a customers table query for each order we rendered. ❌SELECT "customers".* FROM "customers" WHERE "customers"."id" = $1 LIMIT $2 [["id", 1], ["LIMIT", 1]] SELECT "customers".* FROM "customers" WHERE "customers"."id" = $1 LIMIT $2 [["id", 2], ["LIMIT", 1]] SELECT "customers".* FROM "customers" WHERE "customers"."id" = $1 LIMIT $2 [["id", 3], ["LIMIT", 1]] SELECT "customers".* FROM "customers" WHERE "customers"."id" = $1 LIMIT $2 [["id", 4], ["LIMIT", 1]] SELECT "customers".* FROM "customers" WHERE "customers"."id" = $1 LIMIT $2 [["id", 5], ["LIMIT", 1]] SELECT "customers".* FROM "customers" WHERE "customers"."id" = $1 LIMIT $2 [["id", 6], ["LIMIT", 1]] SELECT "customers".* FROM "customers" WHERE "customers"."id" = $1 LIMIT $2 [["id", 7], ["LIMIT", 1]] SELECT "customers".* FROM "customers" WHERE "customers"."id" = $1 LIMIT $2 [["id", 8], ["LIMIT", 1]] SELECT "customers".* FROM "customers" WHERE "customers"."id" = $1 LIMIT $2 [["id", 9], ["LIMIT", 1]] SELECT "customers".* FROM "customers" WHERE "customers"."id" = $1 LIMIT $2 [["id", 10], ["LIMIT", 1]] SELECT "customers".* FROM "customers" WHERE "customers"."id" = $1 LIMIT $2 [["id", 11], ["LIMIT", 1]] SELECT "customers".* FROM "customers" WHERE "customers"."id" = $1 LIMIT $2 [["id", 12], ["LIMIT", 1]] SELECT "customers".* FROM "customers" WHERE "customers"."id" = $1 LIMIT $2 [["id", 13], ["LIMIT", 1]] SELECT "customers".* FROM "customers" WHERE "customers"."id" = $1 LIMIT $2 [["id", 14], ["LIMIT", 1]] SELECT "customers".* FROM "customers" WHERE "customers"."id" = $1 LIMIT $2 [["id", 15], ["LIMIT", 1]] Rails knows customers aren’t loaded, so it lazy loads (does a database lookup for) each customer on the fly. The query returns 15 orders, which means 15 individual customer queries. This is what I mean by N+1. We have N order records. For every order, we have +1 more query to look up the customer. This is a problem when working with large datasets. Each request to the customers table isn’t necessarily a bottleneck — they’re pretty quick. But they add up. And sometimes it’s not just one association per record. It’s many associations per record. The solution to N+1s We fix this by preloading associations.class OrdersController < ApplicationController def index @orders = Order.includes(:customer).all end end By adding the .includes method with the association we want to preload, Rails ensures that every corresponding customer is loaded. Instead of 15 extra SQL queries, we only have 1 extra query:SELECT "orders".* FROM "orders" SELECT "customers".* FROM "customers" WHERE "customers"."id" IN ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15) [["id", 1], ["id", 2], ["id", 3], ["id", 4], ["id", 5], ["id", 6], ["id", 7], ["id", 8], ["id", 9], ["id", 10], ["id", 11], ["id", 12], ["id", 13], ["id", 14], ["id", 15]] Over time it’s easier to know when you’re introducing an N+1 query, but it’s still easy to miss them. Let’s look at how we can catch N+1s before shipping to production. Before Rails 6.1 The Bullet gem is the tool for discovering N+1s. Bullet keeps an eye out for N+1 queries in your application. From Rails logs to notifying you in Slack, it has many ways to warn you of N+1 queries. Remember, it’s still up to you to listen and fix the N+1s. 😅 Rails 6.1+ If you don’t want to fool with another dependency or know that you’ll just ignore Bullet warning you of N+1s, there’s a more invasive option. Rails 6.1 introduced a mechanism for detecting down N+1s: strict_loading.class Order < ApplicationRecord belongs_to :customer, strict_loading: true end Adding this option to the customer association results in an ActiveRecord::StrictLoadingViolationError raised if Rails detects you’re lazy loading the association. When we encounter this error, it’s clear to us what happened:`Order` is marked for strict_loading. The Customer association named `:customer` cannot be lazily loaded. Adding this option to each association is tedious! Luckily, there’s a way to enable this functionality globally. Applying strict loading in development If you only want an error raised in development, not production, you can enable the option in config/development.rbconfig.active_record.strict_loading_by_default = true I like Aaron Francis’ idea of enforcing strict loading in development only. In our companion article on Laravel’s safety mechanisms, he suggests: Lazy loading relationships does not affect the correctness of your application, merely the performance of it. Ideally, all the relations you need are eager loaded, but if not, it simply falls through and lazy loads the required relationships. For that reason, we recommend disallowing lazy loading in every environment except production. Hopefully, all lazy loads will be caught in local development or testing, but in the rare case that a lazy load makes its way into production, your app will continue to work just fine, if a bit slower. Applying strict loading regardless of the environment If performance is critical and it’s important to raise this error in all environments, including production, apply the option in config/application.rb:config.active_record.strict_loading_by_default = true Asynchronous association destruction Active Record provides a mechanism for deleting associated records when a parent record is deleted. The quickest, simplest way to delete dependent data is by providing the dependent: :delete_all option on the association:class Order < ApplicationRecord has_many :invoices, dependent: :delete_all end Deleting this order will execute a separate, single SQL query to delete all the associated invoices. Sometimes, though, the dependent records have “on delete” callbacks that need to run. Because delete_all uses a SQL query, it’s not going to instantiate the dependent records and run the delete callbacks. To ensure the callbacks are run, we’d change the dependent option to destroy:class Order < ApplicationRecord has_many :invoices, dependent: :destroy end Deleting an order now instantiates each dependent, and associated record and calls #destroy on it. But what happens if there are tens, hundreds, or even thousands of associated records? It means tens, hundreds, or even thousands of object instantiations and SQL calls. This long-running functionality feels like something that could run in a background job. As it turns out, Rails agrees. Changing the dependent: :destroy to dependent: :destroy_async will enqueue a background job to destroy the dependent, associated records. It’s worth noting this only works for associations that do not have a foreign key constraint. For more options for destroying dependent relations, take a look at the Miss Hannigan gem. We have another post on deleting data at scale with Rails if you want to dig in deeper. Fail loudly, fail proudly Active Record methods fail loudly or silently. You can often tell if an Active Record method fails loudly or silently by its name. Most (not all) methods that end with a bang (!) will raise an error. What’s the difference? Loud failures Loud failures raise an error and halt code execution. A good example of this is Active Record’s .find method which locates the database record by ID. Trying to find a record that doesn’t exist raises an ActiveRecord::RecordNotFound error.class OrdersController < ApplicationController def update @order = Order.find(params[:non_existent_id] @order.update(order_params) end end We never get to the update call because the lookup raises an error and halts the request. Silent failures Instead of using .find, replace it with .find_by, which takes a list of attributes to look up instead of an ID.class OrdersController < ApplicationController def update @order = Order.find_by(id: params[:non_existent_id] @order.update(order_params) end end In this case, .find_by returns nil. This means we’ll get to the #update action, but it will try to call update on nil, which will raise the ever-present “undefined method x for nil.” 🥶 Comparing the two In this example, we create an order and notification. Once we’re done creating records, we send a notification email.order = Order.create(order_params) notification = Notification.create(notifiable: order) NotificationEmail.deliver_later(notification) What happens if creating the notification fails validation? We get an instance of Notification back that isn’t in the database. The notification is passed to the NotificationEmail and sent in the background. This block of code doesn’t fail. But when the job runs later, in the background, it fails. The notification can’t be retrieved from the database because it was never saved to the database. 😅 Silently failing If we acknowledge it’s okay for a notification to fail, we can continue using the silent failure, create, and add a boundary:notification = Notification.create(notifiable: order) if notification.valid? NotificationEmail.deliver_later(notification) end Failing loudly! But often times I don’t expect code to fail. If it is failing, it’s a signal of a larger problem that should be addressed. In our example, the background job failing wasn’t the root issue. It was a side effect. If we don’t expect creating a notification to ever fail, failing loudly might be a good option. In this example, failing loudly is done by using create!:notification = Noficiation.create!(notifiable: order) Notification.deliver_later(notification) This juicy code block raises an error if validation fails and makes it easier to track. If the job is failing because the notification wasn’t created, we have to figure out what enqueued the job and why it failed. If an error is raised because the Notification failed to create, the job is never enqueued and we immediately know where to look to fix the problem. The error itself signals what went wrong. Credential protection in Rails I once got a bill from AWS for $20,000+. Want to guess what happened? My private git repo was compromised where I had hard-coded AWS credentials. That day I learned about the importance of keeping credentials out of your code. We often store credentials as environment variables. Whenever we add a new credential, we update all the places our application runs: production, CI, the password manager team shares, etc. What if I told you Rails has a built-in mechanism for securely storing credentials? Rails credentials Instead of keeping environment variables in sync across multiple developers and platforms, we can store our credentials securely in our application. To get started, run the Rails CLI:bin/rails credentials:edit --environment=development The first time this command is run it creates two files: config/credentials/development.yml.enc config/credentials/development.key The first file is an encrypted YAML file. When we ran our command, the file was temporarily decrypted and opened in our editor for us to edit. While the file is decrypted, we can update it to store our credentials in a standard YAML key/value structure:stripe: secret_key: SK1234 publishable_key: PK1234 Once we close the file, it’s re-encrypted. The file can be decrypted only with the key it was encrypted with, which was the second file created. Rails adds the decryption keys to .gitignore, keeping it from accidentally being committed to git. Once our secrets are saved, we can access them in the app using the following:Rails.application.credentials.stripe.secret_key Rails.application.credentials.stripe.publishable_key 🚨 Keep the key that Rails generated safe! If you lose it, you’ll lose access to your credentials. If you work on a team, keep it somewhere safe, like a password manager. Multiple environments In the example above, I used the development environment. Rails is going to automatically defer to the development set of credentials in development. I create a credentials file for each environment: development, staging, and production. This makes it easy to keep development and production credentials separate. Development emails in Rails What’s more fun than using real email addresses in development? Accidentally sending emails to those real email addresses in development. 😎 (While we’re here… please don’t send emails from Active Record callbacks.) While tools exist to make this experience better (Letter Opener or Mailtrap), Rails provides a few mechanisms to help. Turn off email delivery This is the least exciting yet most effective mechanism for preventing unwanted emails from being sent. To turn off email delivery, add the following setting to config/development.rb:config.action_mailer.perform_deliveries = false This option may be suitable if you have email previews and a solid test suite. Change the delivery method Maybe you want to have a record of the email being sent for debugging. Action Mailer can save “sent” emails as files instead of delivering them. To do this, add the following setting to config/development.rb:config.action_mailer.deliver_method = :file Any emails “sent” from the application in development will be saved to tmp/mails. This saves the raw output, which might be enough for your use case. For me, though, I typically want to see the email in its final, table-loaded, CSS-less, 1990s HTML email form. Email interceptors The final option we’ll look at is intercepting all emails sent in development and rerouting them to your email address. This only works if you have an active SMTP server or email provider configured in development mode. We do this by defining an email interceptor:class DevelopmentEmailInterceptor def self.delivering_email(message) message.to = ["jason@example.com"] end end The email interceptor implements the .delivering_email method. Inside the method, we’ll change the message’s recipient to our email address. To wire up the interceptor, we add the following option to config/development.rb:config.action_mailer.interceptors = ["DevelopmentEmailInterceptor"] Now, any email sent from development will be rerouted to our email, no matter who it was initially addressed to. Stay safe These tools give us more confidence in building our applications. Having Rails raise an error every time you forget to include an association may feel like a minor annoyance. But it’s less annoying than having to revisit code a few months later to fix a performance issue a preload would have avoided. Having to add the boilerplate strong parameters requires may feel boring. But give me boring over the excitement of a bad actor creating a security incident for my customers. Go enjoy the vast magic of Rails, my friends.]]> Building a multi-region Rails application with PlanetScale https://planetscale.com/blog/rails-multi-region-database 2022-12-08T17:30:00.000Z 2022-12-08T17:30:00.000Z Mike Coutermarsh username: root password: socket: /tmp/mysql.sock development: primary: <<: *default database: multi_region_rails_development primary_replica: <<: *default database: multi_region_rails_development replica: true test: primary: <<: *default database: multi_region_rails_test primary_replica: <<: *default database: multi_region_rails_test replica: true Add the following to your application_record.rb:# app/models/application_record.rb class ApplicationRecord < ActiveRecord::Base primary_abstract_class connects_to database: { writing: :primary, reading: :primary_replica } end Once Rails is aware of your replica connection, you'll be able to manually query it by wrapping any queries in a block using connected_to(role: :reading).ActiveRecord::Base.connected_to(role: :reading) do books = Book.where(author: "Taylor") # all code in this block will be connected to the replica end Automatic connection switching Manually wrapping every read query would be tedious. Rails has a better way. Automatic connection switching enables Rails to swap between your primary and replica connections as needed. All writes will be directed to the primary. Reads will hit the replica. This is what we need for our application to work well automatically when deployed to different regions. To set this up, run:bin/rails g active_record:multi_db And then uncomment the following lines in application.rb:Rails.application.configure do config.active_record.database_selector = { delay: 2.seconds } config.active_record.database_resolver = ActiveRecord::Middleware::DatabaseSelector::Resolver config.active_record.database_resolver_context = ActiveRecord::Middleware::DatabaseSelector::Resolver::Session end Notice this line: config.active_record.database_selector = { delay: 2.seconds }. It's the key detail that will enable your application to handle reading its own writes. Replication lag and reading your own writes The majority of web requests to most Rails applications are GET requests. These requests read data from your database. POST/PUT/PATCH and DELETE requests update data in your application. When using multiple database connections, one common pitfall is replication lag. When using database replicas, there will always be a small delay between when data is written to the primary and when it is available on the replicas. This is known as replication lag. It can vary based on how busy the primary database is. Replication lag becomes a problem for your application when a user writes to the database and then immediately tries to read that same data from the replica. It's possible the data is not there yet and the user will be served an error rather than the data they are expecting. To solve this, Rails has middleware that will automatically set a cookie for 2 seconds after each write. While this cookie is present Rails will direct all reads to the primary rather than the replica. Connecting to the nearest database replica Now that our application can connect to our replica, we need it to selectively connect to the closest one to take advantage of the low latency. To do this, we need to tell our application which set of credentials to use based on where our Rails application is deployed. In this example, we have our connection details stored in Rails credentials.<% # Our application has a region environment variable. # We check this variable and connect to the closest DB region. region = ENV["APP_REGION"] # When in Frankfurt, we use our Frankfurt region. # When in São Paolo, => São Paolo region. region_replica_mapping = { "fra" => Rails.application.credentials.planetscale_fra, "gra" => Rails.application.credentials.planetscale_gra } # If no specific region exists, we’ll connect to the primary. db_replica_creds = region_replica_mapping[region] || Rails.application.credentials.planetscale %> production: primary: <<: *default username: <%= Rails.application.credentials.planetscale&.fetch(:username) %> password: <%= Rails.application.credentials.planetscale&.fetch(:password) %> database: <%= Rails.application.credentials.planetscale&.fetch(:database) %> host: <%= Rails.application.credentials.planetscale&.fetch(:host) %> ssl_mode: <%= Trilogy::SSL_VERIFY_IDENTITY %> primary_replica: <<: *default username: <%= db_replica_creds.fetch(:username) %> password: <%= db_replica_creds.fetch(:password) %> database: <%= db_replica_creds.fetch(:database) %> host: <%= db_replica_creds.fetch(:host) %> ssl_mode: <%= Trilogy::SSL_VERIFY_IDENTITY %> replica: true Once this is in place, we can now have our globally deployed app read data from our globally deployed database. This will result in much faster GET requests for anyone in that region. Any writes will still go to the primary.]]> Secure your connection string with AWS KMS https://planetscale.com/blog/secure-your-connection-string-with-aws-kms 2022-12-07T15:00:00.000Z 2022-12-07T15:00:00.000Z Brian Morrison II ".zip file". In the next modal, click the "Upload" button and select the zip file from your computer. Click "Save" once you’ve selected it. Next, you’ll need to change the default handler from hello to main, which is the name of the binary that was built for Lambda. Under Runtime settings, click "Edit". Change the Handler field to “main” and click "Save". Next, select the "Configuration" tab > "Environment variables" > "Edit". Create an entry named “DSN” and paste in the connection string for your PlanetScale database. You can find this in your PlanetScale dashboard by clicking "Connect", clicking the "Connect with" dropdown, and selecting "Go". Once you have it, paste it in and click "Save". Finally, lets test the function and see if we get data back from the database. Select the "Test" tab, then click the "Test" button. The view should update and display an alert box called Execution result. If you followed all of the previous steps correctly, the box should be green. Expand it and you should see the records from the database under Log output. Now lets see how to encrypt our connection string with a KMS key. Before moving on from Lambda, you’ll need to grab the execution role for this Lambda. You can find that in the "Configuration" tab under "Permissions". Take note of it as you’ll need it in the next step. Create a customer managed key in KMS Start in the AWS console and use the global search to find “key management service”. Select it from the list of available services. If you do not see a button to create a key immediately, select "Customer managed keys" from the left navigation first. Click "Create key". As mentioned earlier, AWS lets you create symmetric and asymmetric keys. Both options can be used to encrypt and decrypt data, but asymmetric keys are useful if you need to download the public key for signing other artifacts outside of AWS. Since we’re only working within AWS, leave "Symmetric" selected and click "Next". In the next view under Alias, give the key a display name for your reference and click "Next". Now you need to configure the key administrators, which can be an IAM user, group, or role. Key administrators are users that are allowed to make changes to the key from the AWS console or APIs. For this tutorial, select your own IAM user account. Scroll down and click "Next". The next view will let you select IAM users, groups, or roles that are allowed to access your key in KMS. Type the name of the execution role for your Lambda function from the previous section and select it from the list. Click "Next" once you’ve selected it. Finally, scroll to the bottom and click "Finish". Encrypt the connection string in Lambda Head back to your Lambda function, select the "Configuration" tab > "Environment variables" > "Edit". Now expand the Encryption configuration section. Check the "Enable helpers for encryption in transit" box and you’ll notice that an Encrypt button is now present next to the DSN environment variable. When you click "Encrypt", a modal will appear where you can select your KMS key created in the previous section. If you expand Decrypt secrets snippet, you’ll also be shown the code you can use to pull in the encrypted value in and decrypt it for use in your code. We’ll be adding this into the Lambda function. Select your KMS key and click "Encrypt". The value for the DSN environment variable should have updated to an encrypted value. Click "Save". Now if you try to test the code again, it should fail since the code doesn't know what to do with the encrypted connection string. Notice how the error is specifically around how the MySQL driver can’t figure out how to connect to the PlanetScale database. To fix this, open main.go again on your computer and update the first half of the file (up through GetDatabase()) to look like the following. The imports will be updated, the init() function will be added, and the GetDatabase() function will be updated to reflect the DSN variable which holds the decrypted connection string.package main import ( "database/sql" "encoding/json" "log" "os" "encoding/base64" "github.com/aws/aws-lambda-go/lambda" "github.com/aws/aws-sdk-go/aws" "github.com/aws/aws-sdk-go/aws/session" "github.com/aws/aws-sdk-go/service/kms" _ "github.com/go-sql-driver/mysql" ) // Set up variables to be used with the encrypted connection string. var functionName string = os.Getenv("AWS_LAMBDA_FUNCTION_NAME") var encrypted string = os.Getenv("DSN") var DSN string // The init function will run first, decrypting DNS into the above variable. func init() { kmsClient := kms.New(session.New()) decodedBytes, err := base64.StdEncoding.DecodeString(encrypted) if err != nil { panic(err) } input := &kms.DecryptInput{ CiphertextBlob: decodedBytes, EncryptionContext: aws.StringMap(map[string]string{ "LambdaFunctionName": functionName, }), } response, err := kmsClient.Decrypt(input) if err != nil { panic(err) } DSN = string(response.Plaintext[:]) } // The Recipe model will hold the data for a record pulled from the database. type Recipe struct { Id int Name string EstTimeToMake int Description string } // Sets up the connection to the PlanetScale database. func GetDatabase() (*sql.DB, error) { db, err := sql.Open("mysql", DSN) // ← Update the second parameter here return db, err } // remainder of the code... Now follow the process from the previous section to build the project, zip it up, and upload it into AWS. Once you do so, test the function again in AWS and it should return data successfully. Conclusion If you’ve followed along, you should have a good understanding on how KMS can be used to encrypt sensitive info within an application build on AWS. This is a much more secure way to store connection strings so that even if your AWS account is compromised, unauthorized users would not be able to access your PlanetScale database. While the examples here used Go, the same principles apply to any application, regardless of the language.]]> All of the tech PlanetScale Vitess replaces https://planetscale.com/blog/all-the-tech-planetscale-replaces 2022-11-30T13:00:00.000Z 2022-11-30T13:00:00.000Z Brian Morrison II PlanetScale and HIPAA https://planetscale.com/blog/planetscale-and-hipaa 2022-11-18T17:03:57.138Z 2022-11-18T17:03:57.138Z Sam Kottler One million connections https://planetscale.com/blog/one-million-connections 2022-11-01T00:43:00.000Z 2022-11-01T00:43:00.000Z Liz van Dijk MySQL Integers: INT BIGINT and more https://planetscale.com/blog/mysql-data-types-integers 2022-10-31T15:00:00.000Z 2022-10-31T15:00:00.000Z Brian Morrison II Announcing Vitess 15 https://planetscale.com/blog/announcing-vitess-15 2022-10-26T15:50:00.000Z 2022-10-26T15:50:00.000Z Vitess Engineering Team What is Vitess: resiliency, scalability, and performance https://planetscale.com/blog/what-is-vitess 2022-10-21T15:00:00.000Z 2022-10-21T15:00:00.000Z Brian Morrison II Laravel’s safety mechanisms https://planetscale.com/blog/laravels-safety-mechanisms 2022-10-19T00:03:57.138Z 2022-10-19T00:03:57.138Z Aaron Francis title . ' - ' . $post->author->name; } This is an example of the N+1 problem! The first line selects all of the blog posts. Then, for every single post, we run another query to get the post’s author.SELECT * FROM posts; SELECT * FROM users WHERE user_id = 1; SELECT * FROM users WHERE user_id = 2; SELECT * FROM users WHERE user_id = 3; SELECT * FROM users WHERE user_id = 4; SELECT * FROM users WHERE user_id = 5; The “N+1“ notation comes from the fact that an additional query is run for each of the n-many records returned by the first query. One initial query plus n-many more. N+1. Even though each individual query is probably quite fast, in aggregate, you can see a huge performance penalty. And because each individual query is fast, this isn’t something that would show up in your slow query log! With Laravel, you can use the preventLazyLoading method on the Model class to disable lazy loading altogether. Problem solved! Truly, it is that simple. You can add the method in your AppServiceProvider:use Illuminate\Database\Eloquent\Model; public function boot() { Model::preventLazyLoading(); } Every attempt to lazy load a relationship will now throw a LazyLoadingViolationException exception. Instead of lazy loading, you’ll need to explicitly eager load your relationships.// Eager load the `author` relationship. $posts = Post::with('author')->get(); foreach($posts as $post) { // `author` is already loaded. echo $post->title . ' - ' . $post->author->name; } Lazy loading relationships does not affect the correctness of your application, merely the performance of it. Ideally, all the relations you need are eager loaded, but if not, it simply falls through and lazy loads the required relationships. For that reason, we recommend disallowing lazy loading in every environment except production. Hopefully, all lazy loads will be caught in local development or testing, but in the rare case that a lazy load makes its way into production, your app will continue to work just fine, if a bit slower. To prevent lazy loading in non-production environments, you can add this to your AppServiceProvider:use Illuminate\Database\Eloquent\Model; public function boot() { // Prevent lazy loading, but only when the app is not in production. Model::preventLazyLoading(!$this->app->isProduction()); } If you want to log errant lazy loading in production, you can register your own lazy load violation handler using the static handleLazyLoadingViolationUsing method on the Model class. In the example below, we will disallow lazy loading in every environment, but in production, we log the violation rather than throwing an exception. This ensures that our application continues to work as intended, but we can go back and fix our lazy load mistakes.use Illuminate\Database\Eloquent\Model; public function boot() { // Prevent lazy loading always. Model::preventLazyLoading(); // But in production, log the violation instead of throwing an exception. if ($this->app->isProduction()) { Model::handleLazyLoadingViolationUsing(function ($model, $relation) { $class = get_class($model); info("Attempted to lazy load [{$relation}] on model [{$class}]."); }); } } Partially hydrated model protection In almost every book about SQL, one of the performance recommendations that you’ll see is to “select only the columns that you need.” It’s good advice! You only want the database to fetch and return the data that you’re actually going to use because everything else is simply discarded. Until recently, this has been a tricky (and sometimes dangerous!) recommendation to follow in Laravel. Laravel’s Eloquent models are an implementation of the active record pattern, where each instance of a model is backed by a row in the database. To retrieve the user with an ID of 1, you can use Eloquent’s User::find() method, which runs the following SQL query:SELECT * FROM users WHERE id = 1; Your model will be fully hydrated, meaning that every column from the database will be present in the in-memory model representation:$user = User::find(1); // -> SELECT * FROM users where id = 1; // Fully hydrated model, every column is present as an attribute. // App\User {#5522 // id: 1, // name: "Aaron", // email: "aaron@example.com", // is_admin: 0, // is_blocked: 0, // created_at: "1989-02-14 08:43:00", // updated_at: "2022-10-19 12:45:12", // } Selecting all of the columns, in this case, is probably fine! But if your users table is extremely wide, has LONGTEXT or BLOB columns, or you’re selecting hundreds or thousands of rows, you probably want to limit the columns to just the ones you plan on using. (Watch our schema videos to learn more about the LONGTEXT and BLOB columns and why you should avoid selecting them if you don't need them.) You can control which columns are selected using the select method, which leads to a partially hydrated model. The in-memory model contains a subset of attributes from the row in the database.$user = User::select('id', 'name')->find(1); // -> SELECT id, name FROM users where id = 1; // Partially hydrated model, only some attributes are present. // App\User { // id: 1, // name: "Aaron", // } Here’s where things get dangerous. If you access an attribute that was not selected from the database, Laravel simply returns null. Your code will think an attribute is null, but really it just wasn’t selected from the database. It might not be null at all! In the following example, a model is partially hydrated with only id and name, then the is_blocked attribute is accessed further down. Because is_blocked was never selected from the database, the attribute’s value will always be null, treating every blocked user as if they aren’t blocked.// Partially hydrate a model. $user = User::select('id', 'name')->find(1); // is_blocked was not selected! It will always be `null`. if ($user->is_blocked) { throw new \Illuminate\Auth\Access\AuthorizationException; } This exact example probably (probably) wouldn’t happen, but when data retrieval and usage are spread across multiple files, something like this will happen. There is no warning anywhere that a model is partially hydrated, and as requirements evolve, you may end up accessing attributes that were never loaded. With extreme care and 100% test coverage, you might be able to prevent this from ever happening, but it’s still a loaded gun pointed straight at your foot. For that reason, we’ve recommended never modifying the SELECT statement that populates an Eloquent model. Until now! The release of Laravel 9.35.0 brings us a new safety feature to prevent this from happening. In 9.35.0 you can call Model::preventAccessingMissingAttributes() to prevent accessing attributes that were not loaded from the database. Instead of returning null, an exception will be thrown, and everything will grind to a halt. This is a very good thing. You can enable this new behavior by adding this to your AppServiceProvider:use Illuminate\Database\Eloquent\Model; public function boot() { Model::preventAccessingMissingAttributes(); } Notice that we enabled this protection across the board, regardless of environment! You could enable this protection only in local development, but the most important place for it to be enabled is production. Unlike N+1 protection, preventing access to missing attributes is not a performance issue, it’s an application correctness issue. Enabling it prevents your application from behaving in unexpected and incorrect ways. Accessing attributes that weren’t selected could lead to all sorts of catastrophic behavior: Data loss Overwriting data Treating free users as paid Treating paid users as free Sending factually incorrect emails Sending the same email dozens of times The list goes on and on. While throwing exceptions in production is inconvenient, it’s much worse to have silent failures that could lead to data corruption. Better to face the exceptions and fix them. Attribute typos and renamed columns This is a continuation of the previous section and another plea to turn on Model::preventAccessingMissingAttributes() in your production environments. We just spent a long time looking at how preventAccessingMissingAttributes() protects you from partially hydrated models, but there are two other scenarios where this method can protect you! The first is typos. Continuing with the is_blocked scenario from above, if you accidentally misspell “blocked,” Laravel will just return null instead of letting you know about your mistake.// Fully hydrated model. $user = User::find(1); // Oops! Spelled "blocked" wrong. Everyone gets through! if ($user->is_blokced) { throw new \Illuminate\Auth\Access\AuthorizationException; } This particular example would likely be caught in testing, but why risk it? The second scenario is renamed columns. If your column started out named blocked and then later you decide it makes more sense for it to be named is_blocked, you’d need to make sure to go back through your code and update every reference to blocked. And if you miss one? It just becomes null.// Fully hydrated model. $user = User::find(1); // Oops! Used the old name. Everyone gets through! if ($user->blocked) { throw new \Illuminate\Auth\Access\AuthorizationException; } Turning on Model::preventAccessingMissingAttributes() would turn this silent failure into an explicit one. Mass assignment protection A mass assignment is a vulnerability that allows users to set attributes that they shouldn’t be allowed to set. For example, if you have an is_admin property, you don’t want users to be able to arbitrarily upgrade themselves to an admin! Laravel prevents this by default, requiring you to explicitly allow attributes to be mass assigned. In this example, the only attributes that can be mass assigned are name and email.class User extends Model { protected $fillable = [ 'name', 'email', ]; } It doesn’t matter how many attributes you pass in when creating or saving the model. Only name and email will get saved:// It doesn’t matter what the user passed in, only `name` // and `email` are updated. `is_admin` is discarded. User::find(1)->update([ 'name' => 'Aaron', 'email' => 'aaron@example.com', 'is_admin' => true ]); Many Laravel developers opt to turn off mass assignment protection altogether and rely on request validation to exclude attributes. That’s totally reasonable! You just need to ensure you never pass $request->all() into your model persistence methods. You can add this to your AppServiceProvider to turn off mass assignment protection altogether.use Illuminate\Database\Eloquent\Model; public function boot() { // No mass assignment protection at all. Model::unguard(); } Remember: you’re taking a risk when you unguard your models! Be sure to never blindly pass in all of the request data.// Only update `name` and `email`. User::find(1)->update($request->only(['name', 'email'])); If you decide to keep mass assignment protection on, there is one other method that you’ll find helpful: the Model::preventSilentlyDiscardingAttributes() method. In the case where your fillable attributes are only name and email, and you try to update birthday, then birthday will be silently discarded with no warning.// We’re trying to update `birthday`, but it won’t persist! User::find(1)->update([ 'name' => 'Aaron', 'email' => 'aaron@example.com', 'birthday' => '1989-02-14' ]); The birthday attribute gets thrown away because it’s not fillable. This is mass assignment protection in action, and it’s what we want! It’s just a little bit confusing because it’s silent instead of explicit. Laravel now provides a way to make that silent error explicit:use Illuminate\Database\Eloquent\Model; public function boot() { // Warn us when we try to set an unfillable property. Model::preventSilentlyDiscardingAttributes(); } Instead of silently discarding the attributes, a MassAssignmentException will be thrown, and you’ll immediately know what’s happening. This protection is very similar to the preventAccessingMissingAttributes protection. It is primarily about application correctness versus application performance. If you’re expecting that data is saved, but it is not saved, that’s an exception and should never be silently ignored, regardless of environment. For that reason, we recommend keeping this protection on in all environments!use Illuminate\Database\Eloquent\Model; public function boot() { // Warn us when we try to set an unfillable property, // in every environment! Model::preventSilentlyDiscardingAttributes(); } Model strictness Laravel 9.35.0 provides a helper method called Model::shouldBeStrict() that controls the three Eloquent “strictness” settings: Model::preventLazyLoading() Model::preventSilentlyDiscardingAttributes() Model::preventsAccessingMissingAttributes() The idea here is that you could put the shouldBeStrict() call in your AppServiceProvider and turn all three settings on or off with one method call. Let’s quickly recap our recommendations for each setting: preventLazyLoading: Primarily for application performance. Off for production, on locally. (Unless you’re logging violations in production.) preventSilentlyDiscardingAttributes: Primarily for application correctness. On everywhere. preventsAccessingMissingAttributes: Primarily for application correctness. On everywhere. Considering this, if you’re planning on logging lazy loading violations in production, you could configure your AppServiceProvider like this:use Illuminate\Database\Eloquent\Model; public function boot() { // Everything strict, all the time. Model::shouldBeStrict(); // In production, merely log lazy loading violations. if ($this->app->isProduction()) { Model::handleLazyLoadingViolationUsing(function ($model, $relation) { $class = get_class($model); info("Attempted to lazy load [{$relation}] on model [{$class}]."); }); } } If you're not planning on logging lazy load violations (which is a reasonable decision!), then you would configure your settings this way:use Illuminate\Database\Eloquent\Model; public function boot() { // As these are concerned with application correctness, // leave them enabled all the time. Model::preventAccessingMissingAttributes(); Model::preventSilentlyDiscardingAttributes(); // Since this is a performance concern only, don’t halt // production for violations. Model::preventLazyLoading(!$this->app->isProduction()); } Polymorphic mapping enforcement A polymorphic relationship is a special type of relationship that allows many types of parent models to share a single type of child model. For example, a blog post and a user may both have images, and instead of creating a separate image model for each, you can create a polymorphic relationship. This lets you have a single Image model that serves both the Post and User models. In this example, the Image is the polymorphic relationship. In the images table, you’ll see two columns that Laravel uses to locate the parent model: an imageable_type and an imageable_id column. The imageable_type column stores the model type in the form of the fully qualified class name (FQCN), and the imageable_id is the model's primary key.mysql> select * from images; +----+-------------+-----------------+------------------------------+ | id | imageable_id | imageable_type | url | +----+-------------+-----------------+------------------------------+ | 1 | 1 | App\Post | https://example.com/1001.jpg | | 2 | 2 | App\Post | https://example.com/1002.jpg | | 3 | 3 | App\Post | https://example.com/1003.jpg | | 4 | 22001 | App\User | https://example.com/1004.jpg | | 5 | 22000 | App\User | https://example.com/1005.jpg | | 6 | 22002 | App\User | https://example.com/1006.jpg | | 7 | 4 | App\Post | https://example.com/1007.jpg | | 8 | 5 | App\Post | https://example.com/1008.jpg | | 9 | 22003 | App\User | https://example.com/1009.jpg | | 10 | 22004 | App\User | https://example.com/1010.jpg | +----+-------------+-----------------+------------------------------+ This is Laravel’s default behavior, but it’s not a good practice to store FQCNs in your database. Tying the data in your database to the particular class name is very brittle and can lead to unforeseen breakages if you ever refactor your classes. To prevent this, Laravel gives us a way to control what values end up in the database with the Relation::morphMap method. Using this method, you can give every morphed class a unique key that never changes, even if the class name does change:use Illuminate\Database\Eloquent\Relations; public function boot() { Relation::morphMap([ 'user' => \App\User::class, 'post' => \App\Post::class, ]); } Now we’ve broken the association between our class name and the data stored in the database. Instead of seeing \App\User in the database, we’ll see user. A good start! We’re still exposed to one potential problem, though: this mapping is not required. We could create a new Comment model and forget to add it to the morphMap, and Laravel will default to the FQCN, leaving us with a bit of a mess.mysql> select * from images; +----+-------------+-----------------+------------------------------+ | id | imageable_id | imageable_type | url | +----+-------------+-----------------+------------------------------+ | 1 | 1 | post | https://example.com/1001.jpg | | 2 | 2 | post | https://example.com/1002.jpg | | .. | ... | .... | . . . . . . . . . . . . . . | | 10 | 22004 | user | https://example.com/1010.jpg | | 11 | 10 | App\Comment | https://example.com/1011.jpg | | 12 | 11 | App\Comment | https://example.com/1012.jpg | | 13 | 12 | App\Comment | https://example.com/1013.jpg | +----+-------------+-----------------+------------------------------+ Some of our imageable_type values are correctly decoupled, but because we forgot to map the App\Comment model to a key, the FQCN still ends up in the database! Laravel has our back (again) by providing us a method to enforce that every morphed model is mapped. You can change your morphMap call to an enforceMorphMap call, and the fall-through-to-FQCN behavior is disabled.use Illuminate\Database\Eloquent\Relations; public function boot() { // Enforce a morph map instead of making it optional. Relation::enforceMorphMap([ 'user' => \App\User::class, 'post' => \App\Post::class, ]); } Now, if you try to use a new morph that you haven’t mapped, you’ll be greeted with a ClassMorphViolationException, which you can fix before the bad data makes it to the database. The most pernicious failures are the silent ones; it’s always better to have explicit failures! Preventing stray HTTP requests While testing your application, it’s common to fake outgoing requests to third parties so you can control the various testing scenarios and not spam your providers. Laravel has offered us a way to do that for a long time by calling Http::fake(), which fakes all outgoing HTTP requests. Most often, though, you want to fake a specific request and provide a response:use Illuminate\Support\Facades\Http; // Fake GitHub requests only. Http::fake([ 'github.com/*' => Http::response(['user_id' => '1234'], 200) ]); In this scenario, outgoing HTTP requests to any other domain will not be faked and will be sent out as regular HTTP requests. You may not notice this until you realize that specific tests are slow or you start hitting rate limits. Laravel 9.12.0 introduced the preventStrayRequests method to protect you from making errant requests.use Illuminate\Support\Facades\Http; // Don’t let any requests go out. Http::preventStrayRequests(); // Fake GitHub requests only. Http::fake([ 'github.com/*' => Http::response(['user_id' => '1234'], 200) ]); // Not faked, so an exception is thrown. Http::get('https://planetscale.com'); This is another good protection to always enable. If your tests need to reach external services, you should explicitly allow that. If you have a base test class, I recommend putting it in the setUp method of that base class:protected function setUp(): void { parent::setUp(); Http::preventStrayRequests(); } In any tests where you need to allow non-mocked requests to go out, you can re-enable that by calling Http::allowStrayRequests() in that particular test. Long-running event monitoring These last few methods aren’t about preventing discrete, incorrect behaviors but rather monitoring the entire application. These methods can be helpful if you don’t have an application performance monitoring tool. Long database queries Laravel 9.18.0 introduced the DB::whenQueryingForLongerThan() method, which allows you to run a callback when cumulative runtime across all of your queries exceeds a certain threshold.use Illuminate\Support\Facades\DB; public function boot() { // Log a warning if we spend more than a total of 2000ms querying. DB::whenQueryingForLongerThan(2000, function (Connection $connection) { Log::warning("Database queries exceeded 2 seconds on {$connection->getName()}"); }); } If you want to run a callback when a single query takes a long time, you can do that with a DB::listen callback.use Illuminate\Support\Facades\DB; public function boot() { // Log a warning if we spend more than 1000ms on a single query. DB::listen(function ($query) { if ($query->time > 1000) { Log::warning("An individual database query exceeded 1 second.", [ 'sql' => $query->sql ]); } }); } Again, these are helpful methods if you do not have an APM tool or a query monitoring tool like PlanetScale’s Query Insights. Request and command lifecycle Similar to long-running query monitoring, you can monitor when your request or command lifecycle takes longer than a certain threshold. Both of these methods are available beginning with Laravel 9.31.0.use Illuminate\Contracts\Http\Kernel as HttpKernel; use Illuminate\Contracts\Console\Kernel as ConsoleKernel; public function boot() { if ($this->app->runningInConsole()) { // Log slow commands. $this->app[ConsoleKernel::class]->whenCommandLifecycleIsLongerThan( 5000, function ($startedAt, $input, $status) { Log::warning("A command took longer than 5 seconds."); } ); } else { // Log slow requests. $this->app[HttpKernel::class]->whenRequestLifecycleIsLongerThan( 5000, function ($startedAt, $request, $response) { Log::warning("A request took longer than 5 seconds."); } ); } } Make the implicit explicit Many of these Laravel safety features take implicit behaviors and turn them into explicit exceptions. In the early days of a project, it’s easy to keep all of the implicit behaviors in your head, but as time goes on, it’s easy to forget one or two of them and end up in a situation where your application is not behaving as you’d expect. You have enough things to worry about. Take some off your plate by enabling these protections!]]> Optimizing queries in arewefastyet https://planetscale.com/blog/arewefastyet-query-optimization-with-insights 2022-10-11T08:00:00.000Z 2022-10-11T08:00:00.000Z Florent Poinsard Harshit Gangal Introduction to MySQL joins https://planetscale.com/blog/introduction-to-mysql-joins 2022-10-07T08:01:46.798Z 2022-10-07T08:01:46.798Z JD Lien Indexing JSON in MySQL https://planetscale.com/blog/indexing-json-in-mysql 2022-10-04T00:03:57.138Z 2022-10-04T00:03:57.138Z Aaron Francis SELECT properties->>"$.request.email" FROM activity_log; +--------------------------------+ | properties->>"$.request.email" | +--------------------------------+ | little.bobby@tables.com | +--------------------------------+ The ->> operator is a shorthand, unquoting extraction operator, making it equivalent to JSON_UNQUOTE(JSON_EXTRACT(column, path)). We could have written the previous SELECT statement using the longhand and gotten the same result.mysql> SELECT JSON_UNQUOTE(JSON_EXTRACT(properties, "$.request.email")) -> FROM activity_log; +-----------------------------------------------------------+ | JSON_UNQUOTE(JSON_EXTRACT(properties, "$.request.email")) | +-----------------------------------------------------------+ | little.bobby@tables.com | +-----------------------------------------------------------+ Which method you choose is a matter of personal preference! Now that we’ve confirmed our expression is valid and accurate, let’s use it to create a generated column.ALTER TABLE activity_log ADD COLUMN email VARCHAR(255) GENERATED ALWAYS as (properties->>"$.request.email"); The first part of the ALTER statement should look very familiar, we’re adding a column named email and defining it as a VARCHAR(255). In the latter half of the statement we declare that the column is generated and that it should always be equal to the result of the expression properties->>"$.request.email". We can confirm our column has been added by selecting it as we would any other column.mysql> SELECT id, email FROM activity_log; +----+-------------------------+ | id | email | +----+-------------------------+ | 1 | little.bobby@tables.com | +----+-------------------------+ You’ll see that MySQL is now maintaining this column for us. If we were to update the JSON value, the generated column value would change as well. Now that we have our generated column in place, we can add an index to it like we would any other column.ALTER TABLE activity_log ADD INDEX email (email) USING BTREE; That’s it! You’ve now indexed the request.email key in your JSON properties column. Let’s verify that MySQL would use the index to speed up queries that are filtering on email.mysql> EXPLAIN SELECT * FROM activity_log WHERE email = 'little.bobby@tables.com'; *************************** 1. row *************************** id: 1 select_type: SIMPLE table: activity_log partitions: NULL type: ref possible_keys: email key: email key_len: 768 ref: const rows: 1 filtered: 100.00 Extra: NULL MySQL reports that it plans to use the email index to satisfy this query. Generated column indexes and the optimizer MySQL's optimizer is a powerful and mysterious entity. When we give MySQL a command, we’re telling it what we want, not how to get it. Often times MySQL will take our query and rewrite it slightly, which is a good thing! Tens of thousands of hours across dozens of years have gone into making the optimizer effective and efficient. When it comes to indexes on generated columns, the optimizer can "see through" different access patterns to ensure the underlying index is being used. We defined an index on email, which is a generated column based on the expression properties->>"$.request.email". We’ve already proven that the index is used when we query against the email column. What’s more interesting is that the optimizer is smart enough to help us out if we forget to query against the named email column! In the following query, we don’t access the generated column by name, but instead use the shorthand JSON extraction operator. (Some rows omitted from the EXPLAIN statement for brevity.)mysql> EXPLAIN SELECT * FROM activity_log -> WHERE properties->>"$.request.email" = 'little.bobby@tables.com'; *************************** 1. row *************************** id: 1 possible_keys: email key: email key_len: 768 [...]: [...] Even though we didn’t explicitly address the column by name, the optimizer understands that there is an index on a generated column based on that expression and opts to use the index. Thanks optimizer! We can confirm this is the case for the longhand as well.mysql> EXPLAIN SELECT * from activity_log WHERE -> JSON_UNQUOTE( -> JSON_EXTRACT(properties, "$.request.email") -> ) = 'little.bobby@tables.com'; *************************** 1. row *************************** id: 1 possible_keys: email key: email key_len: 768 [...]: [...] Again, the optimizer "reads through" our expression and uses the email index. Not convinced? Let’s take a peek at what the optimizer is doing by running a SHOW WARNINGS after our previous EXPLAIN statement to see the rewritten query.mysql> SHOW WARNINGS; *************************** 1. row *************************** Level: Note Code: 1003 Message: /* select#1 */ select `activity_log`.`id` AS `id`,`activity_log`.`properties` AS `properties`,`activity_log`.`created_at` AS `created_at`,`activity_log`.`email` AS `email` from `activity_log` where (`activity_log`.`email` = 'little.bobby@tables.com') If you look closely, you’ll see that the optimizer has rewritten our query and changed the equality comparison to reference the indexed column. This is especially useful if you're unable to control the access pattern because the query is being issued from a 3rd party package in your codebase, or you're unable to change this part of your code for some other reason. If the underlying expression doesn’t match very closely then the optimizer will not be able to use the index, so be sure to take care when creating your generated column. The MySQL documentation explains the optimizer's use of generated column indexes in further detail. Functional indexes Beginning with MySQL 8.0.13, you're able to skip the intermediate step of creating a generated column and create what is called a "functional index." The MySQL documentation calls these functional key parts. A functional index is an index on an expression rather than a column. Sounds a lot like a generated column, doesn’t it? There’s a reason it sounds similar, and that’s because a functional index is implemented using a hidden generated column! We no longer have to create the generated column, but a generated column is still being created. There are a few gotchas with functional indexes though, especially when it comes to using them for JSON. It would be nice to create our JSON index like this:ALTER TABLE activity_log ADD INDEX email ((properties->>"$.request.email")) USING BTREE; But if you do try that, you get a nasty error:Query 1 ERROR: Cannot create a functional index on an expression that returns a BLOB or TEXT. Please consider using CAST. So what’s going on here? In our earlier examples, we were the ones in charge of creating the generated column and we declared it as a VARCHAR(255), which is easily indexable by MySQL. However, when we use a functional index, MySQL is going to create that column for us based on the data type that it infers. JSON_UNQUOTE returns a LONGTEXT value, which is not able to be indexed. Fortunately, the error message points us in the right direction: we need to cast our value to a type that is not LONGTEXT. Casting using the CHAR function tells MySQL to infer a VARCHAR data type.ALTER TABLE activity_log ADD INDEX email ((CAST(properties->>"$.request.email" as CHAR(255)))) USING BTREE; Now that we’ve added the index, we’ll see if it works by running an EXPLAIN.mysql> EXPLAIN SELECT * FROM activity_log -> WHERE properties->>"$.request.email" = 'little.bobby@tables.com'; *************************** 1. row *************************** id: 1 select_type: SIMPLE table: activity_log partitions: NULL type: ALL possible_keys: NULL key: NULL key_len: NULL ref: NULL rows: 1 filtered: 100.00 Extra: Using where Unfortunately, our index isn’t being considered at all, so we’re not out of the woods yet. Unless otherwise specified, casting a value to a string sets the collation to utf8mb4_0900_ai_ci. The JSON extraction functions, on the other hand, return a string with a utf8mb4_bin collation. Therein lies our problem! Because the collation is mismatched between the query's expression and the stored index, our new functional index isn’t being used. The final step is to explicitly set the collation of the cast to utf8mb4_bin.ALTER TABLE activity_log ADD INDEX email (( CAST(properties->>"$.request.email" as CHAR(255)) COLLATE utf8mb4_bin )) USING BTREE; Rerunning the previous EXPLAIN, we can see that we’re finally in a position to use the functional index.mysql> EXPLAIN SELECT * FROM activity_log -> WHERE properties->>"$.request.email" = 'little.bobby@tables.com'; *************************** 1. row *************************** id: 1 possible_keys: email key: email key_len: 1023 [...]: [...] Clearly functional indexes come with a few pitfalls, some of which are explicit and easy to debug, and some that require a little bit more digging into the documentation. Remember that functional indexes use hidden generated columns under the hood. If you prefer to take control of the generated column yourself (even in MySQL 8.0.13 and later) that’s a perfectly reasonable approach! While direct JSON indexing may not be available in MySQL, indirect indexing of specific keys can cover a majority of use cases. Don’t just stop with JSON, either! You can use generated columns and functional indexes across all types of common, hard to index patterns. Go forth and index with confidence.]]> MySQL data types: VARCHAR and CHAR https://planetscale.com/blog/mysql-data-types-varchar-and-char 2022-09-30T15:00:00.000Z 2022-09-30T15:00:00.000Z Brian Morrison II Debugging database errors with Insights https://planetscale.com/blog/debugging-database-errors-with-insights 2022-09-27T00:03:57.138Z 2022-09-27T00:03:57.138Z Rafer Hazen The MySQL JSON data type https://planetscale.com/blog/the-mysql-json-data-type 2022-09-23T15:00:00.000Z 2022-09-23T15:00:00.000Z Mike Stojan SELECT JSON_EXTRACT(songs, '$[3]') FROM songs; +-----------------------------+ | json_extract(songs, '$[3]') | +-----------------------------+ | "Ghost" | +-----------------------------+ We can also use ->, which is the operator equivalent for JSON_EXTRACT.blog-mysql-json/main> SELECT songs->'$[3]' FROM songs; +-----------------+ | songs -> '$[3]' | +-----------------+ | "Ghost" | +-----------------+ If we need the unquoted result, we can use ->>, which is short for JSON_UNQUOTE(JSON_EXTRACT()).blog-mysql-json/main> SELECT songs->>'$[3]' FROM songs; +------------------+ | songs ->> '$[3]' | +------------------+ | Ghost | +------------------+ If we need to add data to the JSON array, we can use JSON_ARRAY_APPEND or JSON_ARRAY_INSERT to update it.UPDATE songs SET songs = JSON_ARRAY_APPEND(songs, '$', "One last song"); UPDATE songs SET songs = JSON_ARRAY_INSERT(songs, '$[0]', "First song"); For more information on how to use all the different JSON functions, please see MySQL's documentation for the JSON data type and the JSON Function reference. Further learning If you'd like to learn more about data types in MySQL, we have an article on the INT data type and one on the VARCHAR data type that you may find useful. We also have short videos on the following data types: Integers Decimals Strings Binary Strings Long Strings Enums Dates JSON]]> Using the PlanetScale serverless driver with AWS Lambda functions https://planetscale.com/blog/using-the-planetscale-serverless-driver-with-aws-lambda-functions 2022-09-21T14:00:00.000Z 2022-09-21T14:00:00.000Z Brian Morrison II "main" to access the main branch. Now click on "Console" to access the web console of the main branch. Run the following two SQL snippets to create a table and add a few records to it.CREATE TABLE hotels( id INT UNSIGNED PRIMARY KEY AUTO_INCREMENT, name VARCHAR(50) NOT NULL, address VARCHAR(50) NOT NULL, stars FLOAT(2) UNSIGNED ); INSERT INTO hotels (name, address, stars) VALUES ('Hotel California', '1967 Can Never Leave Ln, San Francisco CA, 94016', 7.6), ('The Galt House', '140 N Fourth St, Louisville, KY 40202', 8.0); The serverless driver is currently in beta and needs to be enabled on the database level. To do this, click on the "Settings" tab > "Beta features", and click "Enroll" next to the PlanetScale serverless driver for JavaScript line. By enabling this feature, every new password created will have a different hostname, specifically to endpoints that support accessing your database over HTTP. Now head back to the "Overview" tab and click "Connect". From the Connect modal, select "@planetscale/database" from the dropdown. Note the text in the .env tab as we’ll need to configure these as environment variables in AWS. Set up the Lambda function Start by creating an empty folder on your computer and opening VS Code. Open the integrated terminal and run the following command to initialize the project & install the necessary packages:npm init -y npm install @planetscale/database node-fetch Open the package.json file and add a new entry to the file named “type” and give it a value of “module”.{ "name": "serverless-driver-aws-demo", "version": "1.0.0", "description": "", "main": "index.js", "type": "module", # ◀️ add type here "scripts": { "test": "echo \"Error: no test specified\" && exit 1" }, "keywords": [], "author": "", "license": "ISC", "dependencies": { "@planetscale/database": "^1.3.0", "node-fetch": "^3.2.10" } } Create a file called index.js and add the following code to it.import { Client } from '@planetscale/database' import fetch from 'node-fetch' const db = new Client({ fetch, host: process.env.DATABASE_HOST, username: process.env.DATABASE_USERNAME, password: process.env.DATABASE_PASSWORD }) export async function handler(event) { const conn = db.connection() const results = await conn.execute('SELECT * FROM hotels') console.log(results) } Now we need to get the code into an AWS Lambda function. Log into the AWS console, search for “Lambda”, and select it from the list. Click "Create function". Give the function a name and make sure "Node.js 16.x" is selected under Runtime. Once the function has been created, we need to upload a zipped version of the code we wrote. Zip up the contents of the folder, then in AWS, select "Upload from" > ".zip file". Click the "Upload" button from the modal, select the zipped folder you created, and click "Save". Next, select "Configuration" > "Environment variables", and click "Edit" in the main section of the window to add environment variables. Click "Add environment variable" three times to get three entries and populate the fields using the environment variables gathered from the Connect modal in PlanetScale. Click "Save" once you’ve added them. Now head back to the "Code" tab and click "Test". A modal will appear called Configure test event. Populate the "Event name" field with any arbitrary string (I’ll use “Test”), scroll to the bottom, and click "Save". Now click "Test" again and it will run the function. You should see the output of the results object in a tab of the editor. Build an API with API Gateway Now that you’ve seen how to use the serverless driver for JavaScript in the code, let’s explore the other common query types by re-building the function to support API Gateway, and mapping some of the HTTP methods to those queries like so: HTTP Method Name Query Type get SELECT post INSERT put UPDATE delete DELETE In the following code sample, we’ve pulled out the logic to run the SELECT statement from the previous section into the get() function. We’re also using a switch statement on event.requestContext.http.method to map the request to a different function depending on that HTTP method. Finally, we also added a method to handle a post request so we can add data to the database. Update index.js to match the following code, zip up the contents once again, and upload them into Lambda using the process defined earlier:import { Client } from '@planetscale/database' import fetch from 'node-fetch' const db = new Client({ fetch, host: process.env.DATABASE_HOST, username: process.env.DATABASE_USERNAME, password: process.env.DATABASE_PASSWORD }) export async function handler(event) { const conn = db.connection() switch (event.requestContext.http.method) { case 'GET': return await get(conn, event) case 'POST': return await post(conn, event) default: return { statusCode: 404 } } } async function get(conn, event) { const results = await conn.execute('SELECT * FROM hotels') return { statusCode: 200, headers: { 'Content-Type': 'application/json' }, body: JSON.stringify(results.rows) } } async function post(conn, event) { const { name, address, stars } = JSON.parse(event.body) const res = await conn.execute('INSERT INTO hotels (name, address, stars) VALUES (:name, :address, :stars)', { name, address, stars }) if (res.error) { return { statusCode: 500, headers: { 'Content-Type': 'application/javascript' }, body: JSON.stringify(res.error) } } return { statusCode: 200, headers: { 'Content-Type': 'application/javascript' }, body: JSON.stringify({ id: Number(res.insertId) }) } } Now head into the AWS console and find “API Gateway” using the global search. Click on "Create API" to start the process of building a new instance of API Gateway for the Lambda function we created. To create an HTTP API, click the "Build" button in that section. Click on "Add integration". Then select Lambda as the integration type, and select the Lambda you created in the previous section. Give your API a name as well and click "Next". Under Configure routes, change the Resource path to be /hotels and click "Next". Nothing needs to be changed in the Define stages step, so click "Next". Finally, click "Create" to complete the process. Now grab the Invoke URL from the API you just created, we’ll use this to build some simple tests within VS Code. Back in VS Code, create a new file in the root of your directory called tests.http and populate it with the following. Make sure to replace with what you pulled from API Gateway.@hostname = ### Fetch hotels get {{hostname}}/hotels ### Create hotel post {{hostname}}/hotels Content-Type: application/json { "name": "Orka Sunlife Resort", "address": "Güzgülü Mevkii, Ölüdeniz Cd.", "stars": 4.2 } The VS Code Rest Client plugin should recognize this file and display a small link with "Send Request" above each defined request method. Click the "Send Request" link above the get method and you should receive an array of hotels in a second window pane that will be created automatically. Now test the post method by clicking "Send Request" above that one. You should receive an id field to reflect the ID of the inserted record in PlanetScale. Optionally you can also check the database in PlanetScale using the console to run the following script:SELECT * FROM hotels; This should display the newly created hotel along with the original two added earlier. Now let’s get the put and delete methods working. Update the handler function in the code to reflect the following. Note that the switch statement has been updated to handle those methods.export async function handler(event) { const conn = db.connection() switch (event.requestContext.http.method) { case 'GET': return await get(conn, event) case 'POST': return await post(conn, event) case 'PUT': return await put(conn, event) case 'DELETE': return await del(conn, event) default: return { statusCode: 404 } } } At the end of the file, add the put and del JavaScript methods (we have to use del since delete is a keyword in the JavaScript language). Zip and re-upload the code into AWS after this has been done.async function put(conn, event) { const { id } = event.pathParameters const { name, address, stars } = JSON.parse(event.body) const res = await conn.execute('UPDATE hotels SET name=:name, address=:address, stars=:stars WHERE id=:id', { name, address, stars, id }) if (res.error) { return { statusCode: 500, headers: { 'Content-Type': 'application/javascript' }, body: JSON.stringify(res.error) } } return { statusCode: 200 } } async function del(conn, event) { const { id } = event.pathParameters const res = await conn.execute('DELETE FROM hotels WHERE id=:id', { id }) if (res.error) { return { statusCode: 500, headers: { 'Content-Type': 'application/javascript' }, body: JSON.stringify(res.error) } } return { statusCode: 200 } } Since typically put and delete methods are used on individual records, they are often accompanied by a record ID in the URL. We need to add an API route in API Gateway to handle the URL pattern /hotels/{id}. Navigate to your API in API Gateway again, select "Routes" from the left nav, and click "Create". In the route field, add "/hotels/{id}" and click "Create". Select the new route from the list and click "Attach integration". Select your Lambda function from the list and click "Attach integration" again. Now head back to the tests.http file in VS Code and add the following two requests to the file. Notice the JSON under the put request has each field modified just a bit. An ID of 3 is also at the end of the URL, which is how the Lambda code identifies which record it should update.### Update hotel put {{hostname}}/hotels/3 Content-Type: application/json { "name": "Orka Sunlife Resort Aqua", "address": "Güzgülü Mevkii, Ölüdeniz Cd. Turkey", "stars": 4.3 } ### Delete hotel delete {{hostname}}/hotels/3 Run the put request and it simply returns an OK status, but if you run the get request again, you’ll see that the third entry in the array reflects the updated values we sent int. Finally, run the delete request. Again, it returns an OK status. Run the get again and that third record is removed. For more information on how to use the PlanetScale serverless driver for JavaScript, refer to our documentation portal where we have a detailed overview of when you should use it, as well as an example built with Node and Express that you can run directly on your workstation.]]> Declarative MySQL schemas with Atlas CLI https://planetscale.com/blog/declarative-mysql-schemas-with-atlas-cli 2022-09-16T14:00:00.000Z 2022-09-16T14:00:00.000Z Brian Morrison II Giving your password a name lets you identify the credential set in the PlanetScale dashboard. Take note of the USERNAME, ACCESS HOST URL, and PASSWORD values as you’ll need them in the following section. Next, you’ll need to enter into a shell session with the database to create a table. Run the following command to enter the shell:pscale shell hotels_db main Run the following SQL script to create a table called hotels:CREATE TABLE hotels( id INT UNSIGNED PRIMARY KEY AUTO_INCREMENT, name VARCHAR(50) NOT NULL, address VARCHAR(50) NOT NULL, stars FLOAT(2) UNSIGNED ); Generate the schema definition file Atlas makes it easy to apply a "Database as Code" approach to an existing database by generating a file representing the schema of that database. Before you can do so, you’ll need to craft a connection string so the CLI can properly connect to the PlanetScale database created in the previous section. Use the following format to create your own connection string:"mysql://:@/hotels_db?tls=true" Going forward, this article will use as a reference to the connection string above. To generate a schema file based on the database above, run the following command:atlas schema inspect -u > schema.hcl You should now have a file named schema.hcl in the working directory. If you inspect it, it should look like the following. Note how the outer table node contains a reference to the hotels_db schema, as well as a definition for each column created in the previous section.table "hotels" { schema = schema.hotels_db column "id" { null = false type = int unsigned = true auto_increment = true } column "name" { null = false type = varchar(50) } column "address" { null = false type = varchar(50) } column "stars" { null = true type = float unsigned = true } primary_key { columns = [column.id] } } schema "hotels_db" { charset = "utf8mb4" collate = "utf8mb4_0900_ai_ci" } Modify the schema Modifying the schema simply involves making a change to the schema definition file and applying it with the atlas schema apply command. Let’s add a description column to the hotels table by adding the following snippet between the stars column and the primary_key node:column "description" { null = false type = varchar(100) } Run the apply command using the connection string and a reference to the schema.hcl file.atlas schema apply -u -f schema.hcl Atlas will show you the changes it is about to make to the database upon applying the updated schema. Hit enter on your keyboard to confirm the changes.-- Planned Changes: -- Modify "hotels" table ALTER TABLE `hotels_db`.`hotels` ADD COLUMN `description` varchar(100) NOT NULL Use the arrow keys to navigate: ↓ ↑ → ← ? Are you sure?: ▸ Apply Abort Once changes have been applied, you can inspect the table by using the pscale shell, as described above, and running the following DESCRIBE command:DESCRIBE hotels; Notice how the table contains the description column now. That column was added by Atlas when the schema was applied. Closing remarks Atlas can be an incredible utility to add to your DevOps tool kit. It helps you manage your database as code instead of managing your schema manually with SQL commands. Keeping your database schema under version allows it to have accountability (by configuring Atlas to apply changes on git operations) as well as provides a historical reference to see how your database structure changes over time. One thing to note is that when using Atlas with PlanetScale, you’ll need to make sure you don’t turn on safe migrations, as that will prohibit you from running DDL on production.]]> Build a multi-stage pipeline with PlanetScale and AWS https://planetscale.com/blog/build-a-multi-stage-pipeline-with-planetscale-and-aws 2022-09-13T15:00:00.000Z 2022-09-13T15:00:00.000Z Brian Morrison II ”Create new database”. In the modal, name the database and click ”Create database”. Once the main branch is finished initializing, click on the ”Branches” tab and select the main branch. Click ”Promote a branch to production” and confirm on the modal which will appear. Once the branch has been promoted, click the “Connect” button from the Overview to grab the connection string for later. Select ”Go” from the Connect with dropdown. If your password shows as a bunch of asterisks, click "New password" to generate a new set of credentials. Copy the DSN variable in the .env tab, and paste it in your document for later. Make sure to add the connection string for main to your tracking document. Also make sure to add your org name, which can be found right next to the PlanetScale logo in the upper left of the screen Now head back to Branches and click ”New branch”. Name the new branch dev and click ”Create branch”. Once the branch has initialized, click “Connect” here as well to grab the connection string from the dev branch. Make sure to add the dev connection string to your tracking document. Now select the "Console" tab and run the following script to create a table.CREATE TABLE hotels( id INT UNSIGNED PRIMARY KEY AUTO_INCREMENT, name VARCHAR(50) NOT NULL, address VARCHAR(50) NOT NULL, stars FLOAT(2) UNSIGNED ); Now add some data to the table with this script.INSERT INTO hotels (name, address, stars) VALUES ('Hotel California', '1967 Can Never Leave Ln, San Francisco CA, 94016', 3.8), ('The Galt House', '140 N Fourth St, Louisville, KY 40202', 4); Next, let’s merge our dev branch into main. From the Overview tab of the dev branch, click on "Create deploy request". Once PlanetScale has finished validating the changes, click "Deploy changes" and your schema changes will be applied to the main branch. To validate that the changes have been successfully deployed, we can view the schema of the hotels table from the console. Since this database is new, we need to enable the functionality to use the console on the main branch, which is disabled by default. To do this, head to Settings and check the option for "Allow web console access to production branches". Click "Save database settings". Now go to "Branches" > "main" > "Console", and run the following command.DESCRIBE hotels; You should see the columns that were created in the dev branch, even though you are connected to main. Now that our database and branches are set up, we can move into configuring the necessary AWS services. Set up AWS services In this section, we’ll configure a number of services in AWS: Elastic Container Registry (ECR) to store the Docker images for the environments. We’ll also manually upload the starting images to ECR. Two Lightsail container services will be configured, one for QA and one for Production. Some AWS regions are not supported by Lightsail. To keep things consistent, we’ll be using the us-east-1 region throughout this guide. Before proceeding, make sure you’ve switched to us-east-1 using the region switcher in AWS. Create the Elastic Container Registry In the AWS global search, enter "Elastic Container Registry" and select that option from the results. If you don’t have any registries created, click on "Get Started". Otherwise, click "Create repository". In the Create repository form, set the Visibility settings to "Private" and give the repository a name. I’ll be using bookings-api for the repository name. Scroll to the bottom and click "Create repository." You should be redirected to your list of repositories. Grab the URI from the list and note it in that document. Make sure to add the Repository URI to your tracking document. Open a terminal on your computer. Run the following command to authenticate with your new ECR repository, replacing . You should receive a message stating Login Succeeded.aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin The above command requires the AWS CLI. If you do not have it installed, follow the directions provided on AWS’s guide to installing the CLI. Pushing to the ECR repository Before we can deploy containers to Lightsail, we need to get an image of our container into ECR. If you haven’t done so yet, clone the forked repository to your computer. Then open a terminal in the project folder. Run the following commands to build & push the image to ECR, replacing the variable with the URI pulled from ECR.# Build the image docker build --platform=linux/amd64 -t bookings-api . # Tag the 'qa' image to be pushed to ECR docker tag bookings-api:latest :qa # Push the 'qa' image to ECR docker push :qa # Tag the 'prod' image to be pushed to ECR docker tag bookings-api:latest :prod # Push the 'prod' image to ECR docker push :prod Now head back to the AWS console and open the repository you created earlier. If all was successful, you should see images tagged as prod and qa in the list. Create the Lightsail instances Use the AWS global search to find "Lightsail", and open it from the list of results. You’ll be redirected to a completely different UI from the standard AWS console, which is to be expected. In the Lightsail dashboard, select the "Containers" tab, then "Create container service". Make sure that the Container service location is set to Virginia, all zones (us-east-1). If not, click "Change AWS region" and select it from the list of available regions. Under Choose the power, select the "Na" option to keep the cost as low as possible. Skip the Set up your first deployment section for now. We need to configure access to our ECR instance before we can do that. Scroll down to Identify your service and give your service a name. I’ll name mine bookings-api-service-qa. Click "Create container service" to finish the setup. Once the container service has been created, you’ll be dropped into the dashboard for the container service. Select the Images tab, scroll to the bottom, and click "Add repository" under Amazon ECR private repositories. Select the bookings-api repository from the list and click "Add". If the "Add" button is disabled, it is likely because there is a pending operation for your container service. Check the Status field in the header to make sure nothing is currently being done with the container service. Lightsail will start provisioning the proper security permissions to permit itself to access the private ECR instance. This may take a few minutes. Next, select the Deployments tab and click "Create your first deployment". Complete the form like so: Container name: bookings-api-qa Image: :qa Environment variables: LISTEN: 0.0.0.0:80 DSN: Open ports: 80: HTTP In the Public endpoint section, select "bookings-api-qa". Finally, click "Save and deploy". You can monitor the deployment status from the following page. Once the status has changed to Running, you can click the URL next to Public domain to validate that things are working properly. You should receive a simple text response that says “welcome”. Add /hotels to the end of the URL and we should see the data from the PlanetScale database (specifically the dev branch). Now you’ll need to essentially repeat these same steps (create a container service, create an image, and set up the deployment) for the production branch. The main difference will be when setting up the deployment. When creating the container service, set the name to bookings-api-service-prod. Make sure to permit access to your ECR instance before configuring the deployment. You’ll also need to make sure to update the image tag and the DSN environment variable with the connection string from the main branch of the PlanetScale database. Generate a Docker Hub Token Now let’s take a detour and talk Docker Hub. Since Docker Hub limits image pulls based on IP address by default, the chances of our automated system in AWS having that limit already hit are pretty high since it’s a shared environment. In order to bypass this, we’ll need to generate a token for our user account and use that during the build process in AWS. Log into Docker Hub and navigate to "Account Settings". Now select "Security" from the left nav, then "New Access Token". In the modal, give the access token a description and set the permissions to "Public Repo Read-only". This limits what the token can actually be used for with your account. Click "Generate" to get the token. Take note of this token as it won’t be able to be retrieved again (although you can pretty easily delete this one and create another). Once you are done, you can exit Docker Hub as we won’t need to come back. Make sure to add the Docker Hub token and your username to your tracking document Build the pipeline in AWS Now that the resources to host the API are configured in AWS, we can start building the pipeline that will handle both deploying new versions of the code into each environment and promoting schema updates to the database in PlanetScale. Here is a list of tasks we will accomplish in this section: Create a CodeBuild project for QA which will build and deploy the container to ECR and Lightsail. Modify the IAM role for QA to give it the necessary permissions to deploy to ECR and Lightsail. Create a CodeBuild project for Production which will do the same as above, as well as merge database changes in PlanetScale. Modify the IAM role for Production to give it the necessary permissions. QA We’ll need to perform most of the following steps for both the QA and the Production builds, but we can start with the QA environment. Start by using the AWS search to find ‘CodeBuild’, and select it from the list of results. Click "Create project" to start building the QA project. We’ll step through the Create build project form section by section as there are a number of things to set up here. Under Project configuration, name the project bookings-api-qa. In the Source section, select "GitHub" from the Source provider dropdown. If you’ve already connected your GitHub account, you’ll be able to select from a list of repositories you own, otherwise, you can connect using the "Connect to GitHub" button. This will step you through connecting AWS CodeBuild to your GitHub account. Once you’ve connected, you’ll get a few more options in this section. Select "Repository in my GitHub" account and use the search under GitHub repository to find the forked version of the code we’ve been using throughout this guide. Set Source version to “qa” since this is the branch we want to build in this project. In the Primary source webhook events, check the box labeled "Rebuild every time a code change is pushed to this repository". This will allow AWS to configure GitHub to notify AWS when a change is made to the QA branch and to trigger a build on it. Under Event type, select "PUSH" from the list, which will set up the webhook in GitHub to only send a message when commits are pushed to the repository. Expand the Start a build under these conditions toggle and add “refs/heads/qa” to the HEAD_REF field. This will tell CodeBuild to only execute this build if a commit was pushed to the qa branch. The Environment section has a number of items that need to be configured. Here is what each of these should be: Environment image: Managed image — Uses a provided AWS container image to build the code. Runtime(s): Standard — The default Standard runtime. Operating system: Amazon Linux 2 — Since the code should target Linux to be built for Lightsail. Image: aws/codebuild/amazonlinux2-x86_64-standard:4.0 — The latest Amazon Linux image. Image version: Always use the latest — Self-explanatory. Environment type: Linux — The standard Linux environment. Privileged: Checked — We’re building a docker container so this needs to be checked. Service role: New service role — Let CodeBuild create a role with the basic permissions for us. Stay in the Environments section and toggle the "Additional configuration" item to get more options for configuring the environment. Most of these options can remain as is, but we need to add a number of environment variables here so that when a build is triggered, AWS has the necessary info to build and deploy our container image. Find the Environment variables section, and add the following variables. Click "Add environment variable" to add more to the list. DOCKER_HUB_TOKEN — The token you retrieved from Docker Hub. DOCKER_HUB_USER — Your Docker Hub username. REPOSITORY_URI — The ECR repository URI. PS_CONN_STR — The PlanetScale connection string of the dev branch. We will be storing these variables as plain text for the purpose of this article. In a real production environment, sensitive credentials should be stored in a more secure system like AWS Secrets Manager Now onto the Buildspec section. This is where we need to define the steps required to build the image. Since we want to handle the QA and Production build steps a bit differently (specifically when it comes to updating the schema in the PlanetScale database), we need to select "Insert build commands" so we can provide build steps that are not stored with the repository. Once you’ve selected that, click on "Switch to editor" to get a larger text box and paste the below code in. Click "Update buildspec" when done.version: 0.2 phases: build: commands: # Setup environment - docker login -u $DOCKER_HUB_USER -p $DOCKER_HUB_TOKEN # Build the project - docker build --platform=linux/amd64 -t bookings-api . - aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin $REPOSITORY_URI - docker tag bookings-api:latest $REPOSITORY_URI:qa - docker push $REPOSITORY_URI:qa # Deploy - | aws lightsail create-container-service-deployment \ --region us-east-1 \ --service-name bookings-api-service-qa \ --containers "{\"bookings-api-qa\":{\"image\":\"$REPOSITORY_URI:qa\",\"environment\":{\"LISTEN\":\"0.0.0.0:80\", \"DSN\":\"$PS_CONN_STR\"},\"ports\":{\"80\":\"HTTP\"}}}" \ --public-endpoint '{"containerName":"bookings-api-qa","containerPort":80,"healthCheck":{"path":"/"}}' The rest of the settings can be left as they are. Scroll to the bottom and click "Create build project". Updating IAM permissions for QA Although we’ve been allowing AWS to manage permissions for us up to this point, there is a bit of manual configuration that needs to be done before we can build and deploy the container to AWS. Here are the permissions that need to be added to the role for each CodeBuild project: Permit CodeBuild to pull from the ECR instance. Permit CodeBuild to create a service deployment in Lightsail. From within the CodeBuild project, select the Build details tab. Scroll to the Environment section and click the link under Service role. You’ll be redirected to the Role definition in IAM. Under Permissions policies, click "Add permissions" > "Create inline policy". In Create policy view, start with the Service section. Search for “Lightsail” and select it from the results. Under Actions, search for “CreateContainerServiceDeployment” and select it from the list of results. Now to limit the boundaries of the policy to ONLY the QA version of our container service, you’ll need to get the ARN of the container service. The only way to do this at the moment is using the AWS CLI. Open a terminal on your computer and run the following command to get a list of the container services and their ARNs:aws lightsail get-container-services --query 'containerServices[*].[containerServiceName,arn]' --region us-east-1 Your output should show the name of the container service followed by its ARN in a JSON structure. Make sure to note the ARNs of both container services in your tracking document Back in IAM, under the Resources section, make sure "Specific" is checked and click "Add ARN". A modal should appear with a field to paste in the ARN we grabbed from the terminal. Paste the qa ARN in and the other fields should automatically populate. Click "Add" to apply the changes. You can collapse the Lightsail section and click "Add additional permissions" to get a blank form to add more permissions. Select Elastic Container Registry from the list of services. Under Actions, select the following: List DescribeImages ListImages Read BatchCheckLayerAvailability BatchGetImage DescribeRepositories GetAuthorizationToken GetDownloadUrlForLayer Tagging TagResource Write CompleteLayerUpload InitiateLayerUpload PutImage UploadLayerPart The Resources section is a bit more straightforward for ECR. Simply click "Add ARN" and populate the region and repository name. Once you are done, click "Review policy" at the bottom of the page. Give your policy a name and finish by clicking "Create policy". Now that everything is set up, let’s run the build. From the CodeBuild project, click on "Start build". Provided everything is set up properly, you should receive a Succeeded status after the build completes. If not, check the logs below to determine if anything is not set up properly. Production Now that QA is set up and ready to go, we need to set up the production pipeline. Since we will also be creating and approving a Deploy Request in PlanetScale, we need to create a service token in PlanetScale first which will allow the CodeBuild project to access our database using the PlanetScale CLI. In PlanetScale at the root of your organization, click the Settings tab, then "Service tokens". Click "New service token" to open the modal to create a service token. Give your token a name and click "Create service token". The name is for your reference and does not affect the token in any way. Your token will be displayed this one time, so make sure to note it down before moving on. Click "Edit token permissions". Now the next page will show the ID of that token. Note that down as you’ll need it to be set in CodeBuild. Click "Add database access" next. Make sure to add the PlanetScale service token and service token ID to your tracking document Now select your database from the dropdown and check the following options: create_deploy_request read_deploy_request approve_deploy_request Click "Save permissions" once you are done. Now we can head back to AWS to configure CodeBuild. Most of the steps from the previous section will be carried over with a few minor tweaks. Start by creating a new CodeBuild project named bookings-api-prod. Under Source, use all the same settings from QA but set the Source version to main to use the main branch from GitHub. Check the box under Primary source webhook events to "Rebuild every time a code change is pushed". Set the Event type to "PULL_REQUEST_MERGED". Pull Requests have the branch data in the BASE_REF field, so expand "Start a build under these conditions" and set BASE_REF to “refs/heads/main”. Use all the same settings for Environment that were used in QA. Expand the additional options and find Environment variables. Add the same variables you did for the QA pipeline with the following changes: PS_CONN_STR — The PlanetScale connection string for the main branch. PS_TOKEN_ID — The PlanetScale service token ID. PS_TOKEN — The PlanetScale service token. PS_ORG — Your PlanetScale org name. Under Buildspec, select "Insert build commands", then expand the editor to paste the following.version: 0.2 phases: build: commands: # Setup environment - docker login -u $DOCKER_HUB_USER -p $DOCKER_HUB_TOKEN - curl -LO https://github.com/planetscale/cli/releases/download/v0.112.0/pscale_0.112.0_linux_amd64.deb - dpkg -i ./pscale_0.112.0_linux_amd64.deb - pscale --version # Build the project - docker build --platform=linux/amd64 -t bookings-api . - aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin $REPOSITORY_URI - docker tag bookings-api:latest $REPOSITORY_URI:prod - docker push $REPOSITORY_URI:prod # Deploy PlanetScale schema changes - | DR_NUM=$(pscale deploy-request create bookings_api dev --service-token $PS_TOKEN --service-token-id $PS_TOKEN_ID --org $PS_ORG --format json | jq '.number' ) DR_STATE=$(pscale deploy-request show bookings_api $DR_NUM --service-token $PS_TOKEN --service-token-id $PS_TOKEN_ID --org $PS_ORG --format json | jq -r '.deployment.state') while [ "$DR_STATE" = "pending" ]; do sleep 5 DR_STATE=$(pscale deploy-request show bookings_api $DR_NUM --service-token $PS_TOKEN --service-token-id $PS_TOKEN_ID --org $PS_ORG --format json | jq -r '.deployment.state') echo "State: $DR_STATE" done if [ "$DR_STATE" = "no_changes" ]; then pscale deploy-request close bookings_api $DR_NUM --service-token $PS_TOKEN --service-token-id $PS_TOKEN_ID --org $PS_ORG else pscale deploy-request deploy bookings_api $DR_NUM --service-token $PS_TOKEN --service-token-id $PS_TOKEN_ID --org $PS_ORG fi # Deploy - | aws lightsail create-container-service-deployment \ --region us-east-1 \ --service-name bookings-api-service-prod \ --containers "{\"bookings-api-prod\":{\"image\":\"$REPOSITORY_URI:prod\",\"environment\":{\"LISTEN\":\"0.0.0.0:80\", \"DSN\":\"$PS_CONN_STR\"},\"ports\":{\"80\":\"HTTP\"}}}" \ --public-endpoint '{"containerName":"bookings-api-prod","containerPort":80,"healthCheck":{"path":"/"}}' Before we move on, let’s take a moment and examine the script directly under # Deploy PlanetScale schema changes. I’ve added a commented version to explain exactly what each line is doing:# This line will create a deploy request from the dev branch, and is outputting JSON. # It’s piping the JSON to `jq`, which is reading the Deploy Request number to the DR_NUM variable. DR_NUM=$(pscale deploy-request create bookings_api dev --service-token $PS_TOKEN --service-token-id $PS_TOKEN_ID --org $PS_ORG --format json | jq '.number' ) # This line grabs the Deploy Request and stores the state in DR_STATE DR_STATE=$(pscale deploy-request show bookings_api $DR_NUM --service-token $PS_TOKEN --service-token-id $PS_TOKEN_ID --org $PS_ORG --format json | jq -r '.deployment.state') # This loop will wait until PlanetScale has finished checking to see if changes can be applied before moving forward. while [ "$DR_STATE" = "pending" ]; do sleep 5 DR_STATE=$(pscale deploy-request show bookings_api $DR_NUM --service-token $PS_TOKEN --service-token-id $PS_TOKEN_ID --org $PS_ORG --format json | jq -r '.deployment.state') echo "State: $DR_STATE" done # Once the state has been updated, we’re going to check the state to decide how to proceed. if [ "$DR_STATE" = "no_changes" ]; then # If the state is "no_changes", close the request without applying changes. pscale deploy-request close bookings_api $DR_NUM --service-token $PS_TOKEN --service-token-id $PS_TOKEN_ID --org $PS_ORG else # If it's anything else, attempt to deploy (merge) the changes into the `main` branch. pscale deploy-request deploy bookings_api $DR_NUM --service-token $PS_TOKEN --service-token-id $PS_TOKEN_ID --org $PS_ORG fi Scroll to the bottom and click "Create build project". Updating Prod IAM permissions Now we need to update the permissions for the role that was created for this project just like we did for QA. Select the Build details tab, find the Environment section, and click the link under Service role. Click "Add permissions" > "Create inline policy". Select "Lightsail" as the service, check "CreateContainerServiceDeployment" under Actions, and set the ARN of the production container service for Lightsail. Click "Add additional permissions" to add the ECR entry. Select "Elastic Container Registry" as the service, add the same list of actions (see below), and set the ARN just as we did in the QA pipeline. Click Review policy once you are finished. List DescribeImages ListImages Read BatchCheckLayerAvailability BatchGetImage DescribeRepositories GetAuthorizationToken GetDownloadUrlForLayer Tagging TagResource Write CompleteLayerUpload InitiateLayerUpload PutImage UploadLayerPart Give your policy a name and click "Create policy". Now head back to CodeBuild and run the new project that was created just to make sure it deploys successfully. You can also monitor the dashboard in PlanetScale to see the deploy request being created and then closed due to no changes needing to be applied to the database. Testing the entire flow Now that everything has been configured and we’ve tested everything manually, it’s time to see this entire thing in action! In this section, we will: Add a new column to the dev branch in our PlanetScale database. Add a new field to the model in the API. Push the code to the qa branch in GitHub, triggering a deployment to QA in AWS. Create and merge a PR to the main branch in GitHub, triggering a deployment to Production in AWS. This will also handle merging the dev branch of our PlanetScale database into main. First log into PlanetScale, navigate to the dev branch of your database, and open the "Console" tab. Run the following commands individually in the console:ALTER TABLE hotels ADD description VARCHAR(400); DESCRIBE hotels; You should see the new column that was added. Now run the following script to populate the new description field for the first hotel.UPDATE hotels SET description = 'Welcome to the Hotel California, such a lovely place (such a lovely place)' WHERE id = 1; Here is the script being run, as well as SELECT statements both before and after the UPDATE statement above. Now we need to make a change to the code. Make sure you're on the qa branch. Open data/hotels.go and update the Hotel type to have a Description field. Make sure the type is *string so it can handle NULL values since we only added a description to one hotel.type Hotel struct { Id int64 Name string Address string Stars float32 Description *string // Add Description field } Scroll down a bit to the FetchHotels method and update the line with rows.Scan and add a ref to that new Description field.func FetchHotels() ([]Hotel, error) { conn, err := GetDbConnection() if err != nil { return nil, errors.Wrap(err, "(FetchHotels) GetConnection") } query := "SELECT * FROM hotels" rows, err := conn.Query(query) if err != nil { return nil, errors.Wrap(err, "(FetchHotels) db.Query") } hotels := []Hotel{} for rows.Next() { var hotel Hotel // Add `&hotels.Description` to the end of the following line, within the parens err = rows.Scan(&hotel.Id, &hotel.Name, &hotel.Address, &hotel.Stars, &hotel.Description) if err != nil { return nil, errors.Wrap(err, "(FetchHotels) rows.Scan") } hotels = append(hotels, hotel) } return hotels, nil } As an example, here is what the diff looks like in VSCode after the changes were made. Now commit the code and push it to the repository. Check with CodeBuild and a build on QA should be in progress, triggered from the commit. Once the build is finished, check in with the QA container service in Lightsail. Provided the status is Running, you can use the Public domain URL to test the changes. Since we’ve updated the FetchHotels function, add /hotels to the end of the URL to see the list of hotels with the new Description field added. You should see the same list of hotels, with the first one having the description we added earlier in this section. Head back into GitHub and create a pull request, comparing qa and main. By default, GitHub will try to create a Pull Request comparing your repository with the upstream PlanetScale version, so make sure to set the base repository to your version. Give your pull request a name and click "Create pull request". Go ahead and merge the pull request. Head back into CodeBuild, and you’ll notice that the bookings-qa-prod project has a new build. Note that the Source version for the build is pr/5, referring to Pull Request #5, which was the PR number that I had created in my repo. Over in the PlanetScale dashboard, you can also see that a Deploy Request was created and deployed from the CodeBuild project automatically. Conclusion While that was certainly a lot of ground to cover in a single article, building a pipeline from the ground up will have many moving parts and there is often quite a bit of configuration to get them all talking properly. The goal was to create a realistic example of how branching in PlanetScale can help speed up development by automating the process of testing and merging changes between two instances of a database. Did you enjoy this article? Do us a favor and share it with someone awesome!]]> TAOBench: Running social media workloads on PlanetScale https://planetscale.com/blog/taobench-running-social-media-workloads-on-planetscale 2022-09-08T16:36:00.000Z 2022-09-08T16:36:00.000Z Liz van Dijk Gated Deployments: addressing the complexity of schema deployments at scale https://planetscale.com/blog/gated-deployments-addressing-the-complexity-of-schema-deployments-at-scale 2022-09-06T21:50:00.000Z 2022-09-06T21:50:00.000Z Shlomi Noach One million queries per second with MySQL https://planetscale.com/blog/one-million-queries-per-second-with-mysql 2022-09-01T18:43:00.000Z 2022-09-01T18:43:00.000Z Jonah Berquist Zero downtime Laravel migrations https://planetscale.com/blog/zero-downtime-laravel-migrations 2022-08-29T14:00:35.694Z 2022-08-29T14:00:35.694Z Holly Guevara Run SQL script files on a PlanetScale database https://planetscale.com/blog/run-sql-script-files-on-a-planetscale-database 2022-08-25T15:56:00.000Z 2022-08-25T15:56:00.000Z Brian Morrison II instead of your default terminal prompt.pscale shell travel_api main Run the show tables command to show that the hotels table was created.SHOW TABLES; You should see this output:+----------------------+ | Tables_in_travel_api | +----------------------+ | hotels | +----------------------+ Now run a SELECT statement on hotels to see the data that was populated.SELECT * FROM hotels; While this was a relatively simple example, imagine a scenario where you need to create and populate an entire schema using just commands. Doing it in this manner can be much simpler than manually entering all these commands in!]]> How product design works at PlanetScale https://planetscale.com/blog/how-product-design-works-at-planetscale 2022-08-22T14:23:00.000Z 2022-08-22T14:23:00.000Z Jason Long {true /* TODO: if anything in deploy queue */ && (

{true /* TODO: if queue length = 1 */ && <>There is a deployment queued to deploy} {false /* TODO: if queue length > 1 */ && ( <>There are {/* TODO: queue length */} deployments queued to deploy )} ({/* TODO: loop over queue, comma-separate links */} #{/* TODO: DR number */} )

)} ) Annotating TODOs in a React component We will often kick off feature development by adding the necessary feature flags and checks to the API and front-end. These flags allow us to enable new features for specific people and teams. Later, the entire company and early-access customers can be included before shipping to everyone. Our employees have a high level of trust with each other and the autonomy to decide how best to approach a problem, implement a solution, and ship it. Because our product designers can code, we can avoid the standard handoff process. In our experience, this results in less friction between teams and a better product for our customers.]]> Introducing the PlanetScale serverless driver for JavaScript https://planetscale.com/blog/introducing-the-planetscale-serverless-driver-for-javascript 2022-08-18T14:31:00.000Z 2022-08-18T14:31:00.000Z Taylor Barnett Matt Robenolt ', username: '', password: '' } Then, once your connection configuration is set, you will connect to and execute a SQL command on PlanetScale.const conn = connect(config) const results = await conn.execute('SHOW TABLES') console.log(results) The driver also handles your SQL sanitization to help prevent security issues like SQL injection. For example, this is useful in queries like the following with a parameter:conn.execute('SELECT * FROM users WHERE email=?', ['foo@example.com']) You can read more about the driver and its features in the PlanetScale serverless driver for JavaScript documentation. Want to try it out? You can check out the example application code from github and run the application to try out these features.In the app, you can choose to have the data pulled from a PlanetScale database using Cloudflare Workers, Vercel Edge Functions, or Netlify Edge Functions. We have separated how each of these functions works. You can see the Cloudflare Workers, Vercel Edge Functions, and Netlify Edge Functions examples in their own subdirectory. Try it out yourself Ready to try out the driver in your serverless and edge compute platform of choice? Get started in the PlanetScale documentation. Tweet at us @planetscale or post in our GitHub Discussion group to share your experience with the new driver.]]> Introducing FastPage: Faster offset pagination for Rails apps https://planetscale.com/blog/fastpage-faster-offset-pagination-for-rails-apps 2022-08-16T14:00:00.000Z 2022-08-16T14:00:00.000Z Mike Coutermarsh 5 Thank you ❤️ This gem was inspired by Hammerstone’s fast-paginate for Laravel and @aarondfrancis’s excellent blog post: Efficient Pagination Using Deferred Joins. We were so impressed with the results, we had to bring this to Rails as well.]]> How to kill Sidekiq jobs in Ruby on Rails https://planetscale.com/blog/how-to-kill-sidekiq-jobs-in-ruby-on-rails 2022-08-15T14:00:00.000Z 2022-08-15T14:00:00.000Z Elom Gomez Database DevOps https://planetscale.com/blog/database-devops 2022-08-08T17:03:57.138Z 2022-08-08T17:03:57.138Z Sam Lambert How PlanetScale prevents MySQL downtime https://planetscale.com/blog/how-planetscale-prevents-mysql-downtime 2022-08-02T14:01:00.000Z 2022-08-02T14:01:00.000Z Sam Lambert Ruby on Rails: 3 tips for deleting data at scale https://planetscale.com/blog/ruby-on-rails-3-tips-for-deleting-data-at-scale 2022-08-01T14:00:00.000Z 2022-08-01T14:00:00.000Z Mike Coutermarsh The Slotted Counter Pattern https://planetscale.com/blog/the-slotted-counter-pattern 2022-07-28T16:34:56.745Z 2022-07-28T16:34:56.745Z Sam Lambert select * from slotted_counters; +----+-------------+-----------+------+-------+ | id | record_type | record_id | slot | count | +----+-------------+-----------+------+-------+ | 1 | 123 | 456 | 2 | 21 | | 2 | 123 | 456 | 52 | 99 | | 3 | 123 | 456 | 55 | 321 | | 4 | 123 | 456 | 0 | 442 | | 7 | 123 | 456 | 48 | 69 | | 8 | 123 | 456 | 20 | 661 | | 9 | 123 | 456 | 56 | 62 | | 10 | 123 | 456 | 18 | 371 | | 11 | 123 | 456 | 22 | 127 | | 12 | 123 | 456 | 58 | 33 | | 13 | 123 | 456 | 23 | 322 | +----+-------------+-----------+------+-------+ 11 rows in set (0.00 sec) Getting the count for record_id 456 is as simple as this SELECT query:SELECT SUM(count) as count FROM slotted_counters WHERE (record_type = 123 AND record_id = 456); Now we can have requests executing counter increments in parallel without causing contention and effecting concurrency. There are a few different ways you can implement this pattern, but it comes down to the architecture of your app. One way would be to query the slotted_counters table to roll up the data and update a column stored with the rest of the data.]]> Behind the scenes: How we built Password Roles https://planetscale.com/blog/behind-the-scenes-how-we-built-password-roles 2022-07-27T14:00:00.000Z 2022-07-27T14:00:00.000Z Phani Raju Safely dropping MySQL tables https://planetscale.com/blog/safely-dropping-mysql-tables 2022-07-25T14:00:00.000Z 2023-10-26T14:00:00.000Z David Graham .ibd file was updated. Or you can check the .frm file for DDL changes, which can give you the last known modification. So, you do have some options to find the last time a table was modified, but the solutions aren't very straightfoward. Doing this each time you want to drop a table could drastically delay your team's speed to production. Using PlanetScale to safely drop MySQL tables At PlanetScale, our mission is to create the most scalable, developer-friendly database platform. Dropping tables is never fun, but we wanted to make the process as stress-free as possible. To accomplish this, we built an in-dashboard feature that checks if tables are truly unused during deploy requests and warns you if the table to be dropped was recently queried. Identifying table usage with Insights On top of warning you, we also want to help you find when and where the table is being queried. If you run into this warning, you can use Insights, our in-dashboard query monitoring tool, to help identify where the table is being queried. With Insights, you can narrow down your analysis to individual query performance. We also surface SQL comments on queries, so you can tag your queries with additional information to track down where they came from. Instrumenting queries with comment tags can help you identify which application is still using the table. Once you remove the query from any remaining applications, you can confidently drop the table. Queries against individual tables can always be found by going to your Insights page and using the table: query syntax in the filter input box, as shown below. This reveals how many dependencies there are on the table before attempting to drop it. Try it out Hopefully this addition will make cleaning up unused tables a little less stressful. For more information about how to use Insights, check out our documentation. We love hearing from you! If you have any questions or feedback, don’t hesitate to contact us.]]> Temporal Workflows at scale with PlanetScale: Part 1 https://planetscale.com/blog/temporal-workflows-at-scale-with-planetscale-part-1 2022-07-22T14:45:00.000Z 2022-07-22T14:45:00.000Z Savannah Longoria Announcing Teams: An easier way to manage database administrator access https://planetscale.com/blog/announcing-teams-an-easier-way-to-manage-database-administrator-access 2022-07-20T15:00:00.000Z 2022-07-20T15:00:00.000Z Iheanyi Ekechukwu We now display PlanetScale system status directly in your dashboard https://planetscale.com/blog/we-now-display-planetscale-system-status-directly-in-your-dashboard 2022-07-19T14:45:00.000Z 2022-07-19T14:45:00.000Z Mike Coutermarsh { const res = await fetch('https://www.planetscalestatus.com/api/v2/incidents/unresolved.json') const json = await res.json() let incident = json?.incidents?.[0] || {} if (incident) { incident = { ...incident, url: `https://www.planetscalestatus.com/incidents/${incident.id}` } } return new Response(JSON.stringify({ ...incident }), { status: 200, headers: { 'content-type': 'application/json', 'cache-control': 's-maxage=1, stale-while-revalidate' } }) } Take note of the cache-control header. This is an important detail that instructs Vercel to serve our users with the response from their cache while updating the cache in the background. This ensures users always get a super fast response, and the data is up-to-date as well. It works perfectly for this use case. The React component In our UI, we hit the edge function to check for any statuses, and then display the most recent one, if available.import React from 'react' import useSWR from 'swr' import { SWR_OPTIONS, fetchSWR } from '@/utils/api' import Icon from './Icon' interface Incident { id: string url: string name: string } const IncidentStatus: React.FC = () => { const { data } = useSWR(['/api/incidents', SWR_OPTIONS], fetchSWR) if (!data?.id) { return null } return (
{data.name} · View status
) } export default IncidentStatus Let’s connect We hope this addition is helpful to your workflow. At PlanetScale, we’re users of our own product, so we’re constantly trying to figure out new ways to improve developer experience, both for you and ourselves. If you have any feedback or questions, we’d love to hear from you. You can contact us or find us on Twitter.]]> How do Database Indexes Work? https://planetscale.com/blog/how-do-database-indexes-work 2022-07-14T15:18:36.310Z 2022-07-14T15:18:36.310Z Justin Gage Getting started with the PlanetScale CLI https://planetscale.com/blog/getting-started-with-the-planetscale-cli 2022-07-12T14:58:00.000Z 2022-07-12T14:58:00.000Z Brian Morrison II is the name of the database you want to create:pscale database create In this article, we’ll create and work with a database called cli-db. MySQL shell Now, we need to drop into a MySQL shell within the database to create a table. To do this, run the following command:pscale shell cli-db Your terminal prompt should change to indicate you are now connected to and running commands in the context of the database we just created. Since this is a new database, we don’t have any tables created yet. Let’s run the following command to create a table that mirrors the Post model from Beam.CREATE TABLE `Post` ( `id` int NOT NULL AUTO_INCREMENT, `title` varchar(255) NOT NULL, `content` text NOT NULL, `contentHtml` text NOT NULL, `hidden` tinyint(1) NOT NULL DEFAULT '0', `createdAt` datetime(3) NOT NULL DEFAULT CURRENT_TIMESTAMP(3), `updatedAt` datetime(3) NOT NULL, `authorId` varchar(191) NOT NULL, PRIMARY KEY (`id`), KEY `Post_authorId_idx` (`authorId`), FULLTEXT KEY `Post_title_content_idx` (`title`,`content`) ); Show tables When you hit enter, you shouldn’t get any output. You can check that the table exists with:SHOW TABLES; Working with Branches On top of managing your databases and tables, you can also manage your branches with the PlanetScale CLI. List all branches To demonstrate this, start by listing your existing branches on the database we created with:pscale branch list cli-db Promote a branch In this example, there is only one branch (main), but it’s currently not flagged as a production branch, so let’s promote it to production using:pscale branch promote cli-db main Now that we have main set as our production branch, let’s create a branch off of main called dev. Run the following command to create that branch:pscale branch create cli-db dev You should get a message stating the branch was created successfully. You can also check the dashboard to verify the branch exists. Now that you have another branch to work on, let’s modify the schema of the dev branch and merge it into the main branch. Drop into a shell again with:pscale shell cli-db Since you have multiple branches, the CLI will ask which branch you want to enter. Select dev and hit enter. Add a new column called tag with the following SQL command in the shell.ALTER TABLE Post ADD tag varchar(255); You can use the DESCRIBE command to view how the table looks now.DESCRIBE Post; As you can see, tag is now added to the schema in dev. Create a deploy request Now let’s merge the changes in the dev branch into main using by creating a new deploy request using the CLI.pscale deploy-request create cli-db dev This creates the deploy request for the cli-db, and we’re stating we want to merge the dev branch into the production branch. List all deploy requests You can show all active deploy requests with:pscale deploy-request list cli-db You can also see the deploy requests in the dashboard using the Deploy requests tab. Merge a deploy request To finish this off, let’s merge this deploy request into the main branch. In your terminal, run the following where 1 is the deploy request number shown from the previous step:pscale deploy-request deploy cli-db 1 Now you can check the schema of your main branch using the MySQL shell from the CLI. Enter into the shell with:pscale shell cli-db Select the main branch from the terminal. Describe the Post table again to verify that our changes are now active. To learn more about the PlanetScale CLI, you can use the CLI Reference page in our docs which lists all of the available commands and how to use them. You can also use pscale --help to list available commands for further help within your terminal.]]> Consensus algorithms at scale: Part 8 - Closing thoughts https://planetscale.com/blog/consensus-algorithms-at-scale-part-8 2022-07-07T15:02:00.000Z 2022-07-07T15:02:00.000Z Sugu Sougoumarane Deploy requests now alert on potential unwanted changes https://planetscale.com/blog/deploy-requests-now-alert-on-potential-unwanted-changes 2022-07-06T15:00:00.000Z 2022-07-06T15:00:00.000Z Mike Coutermarsh Consensus algorithms at scale: Part 7 - Propagating requests https://planetscale.com/blog/consensus-algorithms-at-scale-part-7 2022-07-01T15:00:00.000Z 2022-07-01T15:00:00.000Z Sugu Sougoumarane Identifying slow Rails queries with sqlcommenter https://planetscale.com/blog/identifying-slow-rails-queries-with-sqlcommenter 2022-06-29T15:23:56.168Z 2022-06-29T15:23:56.168Z Mike Coutermarsh Iheanyi Ekechukwu User.first You should see your application name in sqlcommenter format. User Load (0.6ms) SELECT `user`.* FROM `user` ORDER BY `user`.`id` ASC LIMIT 1 /*application='ApiBb'*/ Using annotate If you need even more detail for a specific query, Rails 7 also added the annotate method, which lets you add a comment to a query. For example, the following query will add source='user_metrics_runner' as a comment:[3] pry(main)> User.where(name: "iheanyi").annotate("source='user_metrics_runner'") User Load (0.5ms) SELECT `user`.* FROM `user` WHERE `user`.`name` = 'iheanyi' /* source='user_metrics_runner' */ This is useful in situations where the default query log tags aren’t enough. Using with PlanetScale Query Insights PlanetScale Query Insights, our built-in query debugging and analysis tool, is compatible with sqlcommenter. Any query that takes over 1 second to run will get recorded and tagged with the values you’ve set in your sql comments. For example, here is a slow query from our own application:SELECT schema_snapshot.* FROM schema_snapshot WHERE schema_snapshot.ready = true AND created_at > :created_at AND schema_snapshot.deleted_at IS NULL ORDER BY schema_snapshot.id ASC LIMIT 10000 /*application='ApiBb,job='ScheduleSnapshotJob'*/ Using Insights and tags on the slow query, we were able to find exactly where this query was coming from. This enabled us to quickly find and fix the issue in our application. You can try out Query Insights today by signing up for a PlanetScale account and navigating to "Insights" in the dashboard. If you're using a Rails application, be sure to check out the Rails + PlanetScale quickstart and our Rails sqlcommenter gem. This powerful combination, Rails + sqlcommenter + Insights, can greatly improve your query debugging experience. Learn more Rails Query Logs PlanetScale Query Insights Sqlcommenter activerecord_sql-commenter Rails PlanetScale quickstart]]> Announcing Vitess 14 https://planetscale.com/blog/announcing-vitess-14 2022-06-28T15:50:00.000Z 2022-06-28T15:50:00.000Z Vitess Engineering Team Grouping and aggregations on Vitess https://planetscale.com/blog/grouping-and-aggregations-on-vitess 2022-06-24T14:55:30.336Z 2022-06-24T14:55:30.336Z Andres Taylor Consensus algorithms at scale: Part 6 - Completing requests https://planetscale.com/blog/consensus-algorithms-at-scale-part-6 2022-06-21T17:06:12.714Z 2022-06-21T17:06:12.714Z Sugu Sougoumarane Introducing PlanetScale Insights: Advanced query monitoring https://planetscale.com/blog/introducing-planetscale-insights-advanced-query-monitoring 2022-05-26T14:01:00.000Z 2022-05-26T14:01:00.000Z Holly Guevara Extract, load, and transform your data with PlanetScale Connect https://planetscale.com/blog/extract-load-and-transform-your-data-with-planetscale-connect 2022-05-25T15:00:00.692Z 2022-05-25T15:00:00.692Z James Q Quick Introducing PlanetScale Portals: Read-only regions https://planetscale.com/blog/introducing-planetscale-portals-read-only-regions 2022-05-24T15:00:10.621Z 2022-05-24T15:00:10.621Z Taylor Barnett username: root password: socket: /tmp/mysql.sock development: primary: <<: *default database: multi_region_rails_development primary_replica: <<: *default database: multi_region_rails_development replica: true test: primary: <<: *default database: multi_region_rails_test primary_replica: <<: *default database: multi_region_rails_test replica: true This will allow you to send queries to your read-only region or take advantage of Rails "automatic role switching" to route queries for you.ActiveRecord::Base.connected_to(role: :reading) do books = Book.where(author: "Taylor") # all code in this block will be connected to the read-only region end You can set up your production application to connect to your nearest PlanetScale region for reads. This will result in your app having low-latency reads. In this example, we have our connection details stored in Rails credentials.<% # Our application has a region environment variable. # We check this variable and connect to the closest DB region. region = ENV["APP_REGION"] # When in Frankfurt, we use our Frankfurt region. # When in São Paolo, => São Paolo region. region_replica_mapping = { "fra" => Rails.application.credentials.planetscale_fra, "gra" => Rails.application.credentials.planetscale_gra } # If no specific region exists, we’ll connect to the primary. db_replica_creds = region_replica_mapping[region] || Rails.application.credentials.planetscale %> production: primary: <<: *default username: <%= Rails.application.credentials.planetscale&.fetch(:username) %> password: <%= Rails.application.credentials.planetscale&.fetch(:password) %> database: <%= Rails.application.credentials.planetscale&.fetch(:database) %> host: <%= Rails.application.credentials.planetscale&.fetch(:host) %> ssl_mode: verify_identity primary_replica: <<: *default username: <%= db_replica_creds.fetch(:username) %> password: <%= db_replica_creds.fetch(:password) %> database: <%= db_replica_creds.fetch(:database) %> host: <%= db_replica_creds.fetch(:host) %> ssl_mode: <%= Trilogy::SSL_VERIFY_IDENTITY %> replica: true Once this is in place, we can now have our globally deployed app read data from our globally deployed database. This will result in much faster GET requests for anyone in that region. Any writes will still go to the primary. Automatic role switching and reading your own writes We can take this one step further by having all our read queries hit the read-only region without specifying it in our code. We can also tell Rails to read from our primary if the user recently wrote to the database. This protects our users from ever reading stale data due to replication lag. To do this, we need to set reading/writing roles for our models:# app/models/application_record.rb class ApplicationRecord < ActiveRecord::Base primary_abstract_class connects_to database: { writing: :primary, reading: :primary_replica } end Then we can enable automatic role switching by adding the following to our production config.# config/environments/production.rb config.active_record.database_selector = { delay: 2.seconds } config.active_record.database_resolver = ActiveRecord::Middleware::DatabaseSelector::Resolver config.active_record.database_resolver_context = ActiveRecord::Middleware::DatabaseSelector::Resolver::Session This tells Rails to send all reads to our read-only region and writes to our primary. After each write, it will set a cookie that will send all reads to the primary for 2 seconds, allowing users to read their own writes. (Also, thank you to PlanetScale software engineer, Mike Coutermarsh, for help with the Ruby on Rails code in this section.) Pricing Any database on a Base or Enterprise plan can create read-only database regions. The pricing for Portals is based on storage costs and row reads. Storage costs Your storage costs will increase linearly with the number of read-only regions you purchase. For example, if your production branch is 10GB, each read-only region added will increase your total storage cost by 10GB. Portals’ storage costs are prorated by month. If you added a read-only region to your 10GB branch on the 15th, you’ll get billed for 5GB of usage. Adding new read-only regions will always be billed as standalone storage and will not count toward your included storage. Row reads Queries issued to your read-only region will contribute to your total billable row reads per month. To make it easier to track the cost, your invoice details will show a new line for rows read from any read-only region. Try it out today PlanetScale Portals is available in beta today. You can create a new read-only region in any PlanetScale database on a Base or Enterprise plan. Sign up or log into your PlanetScale account and go to your database’s production branch page to add a region. Read more in the Portals docs. If you have feedback, tweet at us @planetscale or post in our GitHub Discussion group.]]> The operational relational schema paradigm https://planetscale.com/blog/the-operational-relational-schema-paradigm 2022-05-09T17:48:00.000Z 2022-05-09T17:48:00.000Z Shlomi Noach Consensus algorithms at scale: Part 5 - Handling races https://planetscale.com/blog/consensus-algorithms-at-scale-part-5 2022-04-28T15:49:00.000Z 2022-04-28T15:49:00.000Z Sugu Sougoumarane Consensus algorithms at scale: Part 4 - Establishment and revocation https://planetscale.com/blog/consensus-algorithms-at-scale-part-4 2022-04-06T15:19:00.000Z 2022-04-06T15:19:00.000Z Sugu Sougoumarane Generics can make your Go code slower https://planetscale.com/blog/generics-can-make-your-go-code-slower 2022-03-30T00:00:00.000Z 2022-03-30T00:00:00.000Z Vicent Marti Why we chose NanoIDs for PlanetScale’s API https://planetscale.com/blog/why-we-chose-nanoids-for-planetscales-api 2022-03-29T17:30:00.000Z 2022-03-29T17:30:00.000Z Mike Coutermarsh Revert a migration without losing data https://planetscale.com/blog/revert-a-migration-without-losing-data 2022-03-24T12:01:46.798Z 2022-03-24T12:01:46.798Z Taylor Barnett Behind the scenes: How schema reverts work https://planetscale.com/blog/behind-the-scenes-how-schema-reverts-work 2022-03-24T12:00:00.000Z 2022-03-24T12:00:00.000Z Holly Guevara Shlomi Noach How to Prevent SQL Injection Attacks in Node.js https://planetscale.com/blog/how-to-prevent-sql-injection-attacks-in-node-js 2022-03-03T17:27:00.000Z 2022-03-03T17:27:00.000Z James Q Quick { const {userQuery} = req.params; const query = 'SELECT * FROM Repository WHERE TAG = '${userQuery}' AND public = 1'; const [rows] = await connection.query(query); res.json(rows); }); app.listen(3001, () =>{ console.log('App is running'); }); Preventing SQL injection attacks There are a few common ways to prevent SQL injection attacks: Don’t allow multiple statements Use placeholders instead of variable interpolation Validate user input Allowlist user input Don’t allow multiple statements if you can avoid it Conveniently, number 1 is handled by the mysql2 client (and many other database clients). It prevents multiple statements from being executed by default. So, even if the user submits an input that attempts to terminate a query and run a second one, the second one won’t run. This is the default configuration, but you can override that if you choose. Although this configuration property is available, it is typically not recommended to allow multiple statements unless absolutely necessary.const connection = await mysql.createConnection({ uri: process.env.DATABASE_URL, multipleStatements: true }) To emphasize the need for more levels of protection, refer to the example above where injecting a comment (ex. javascript';--) into the SQL allowed the user to read from private repositories. Since that was done using only one statement, setting multipleStatements: false still wouldn’t be enough. Use placeholders Therefore, you should never accept raw input from a user and input it directly into your query string. Instead, you should use placeholders (?) (or parametrized queries) which would look like this (notice the ? as the placeholder):const query = 'SELECT * FROM Repository WHERE TAG = ? AND public = 1' const [rows] = await connection.query(query, [userQuery]) By using placeholders, the malicious SQL will be escaped and treated as a raw string, not as actual SQL code. The end result query would look like this: SELECT * FROM Repository WHERE TAG = `javascript';--` AND public = 1; Thanks to using placeholders, the malicious SQL is not run and instead, is treated as a search query as intended. Input validation In addition to using placeholders, you can add logic in your applications to prevent invalid user input. Let’s stick with the example of querying public repositories by tag. For demo purposes, you can assume that you should not have a tag that includes special characters or numbers. In other words, tags should only use capital and lowercase letters (A-Z, a-z). This means you can add logic to your application to validate that user input matches the correct formatting (no numbers and no special characters). To do this, you can create a regex pattern to match the user input. If it doesn’t match, return an error. app.get('/repositories/:userQuery', async (req, res) => { const {userQuery} = req.params; const onlyLettersPattern = /^[A-Za-z]+$/; if(!userQuery.match(onlyLettersPattern)){ return res.status(400).json({ err: "No special characters and no numbers, please!"}) } ... }); Now the code doesn’t even get to the SQL part unless a valid input is passed. You can apply this method with any sort of validation that is relevant to your data. For example, if you allow the user to query by an id property which should be a number, you can throw an error if the input isn’t a valid number. app.get('/repositories/:id', async (req, res) => { const {id} = req.params; if(isNaN(Number(id))) { return res.status(400).json({ err: "Numbers only, please!"}) } ... Allowlisting One last option you have is to use allowlisting, a specific type of input validation. Allowlisting is useful if you know every possible valid user input. From there, you can easily reject anything else. For example, let’s say for your repository tags, there are only three valid tags: “javascript”, “html”, and “css”. If that’s the case, then you can check whether or not the user input is "allowlisted" by comparing it against known valid inputs. app.get('/repositories/:userQuery', async (req, res) => { const {userQuery} = req.params; const validTags = ["javascript", "html", "css"]; if(!validTags.includes(userQuery)){ return res.status(400).json({err: "Valid tags only, please!"}); } ... }); Yes, this example is a bit simplified with just three valid tags, but this works at scale as well. A more realistic scenario might be that you store all known tags in their own table in your database. Then, to validate the user input, you can check against all the tag records in your database. Wrap up Hopefully, this helped give you a good overview of what SQL injection attacks are and how to prevent them. They can be detrimental to your application and business, so it’s important to plan ahead when accepting user input for your database queries to prevent any negative side effects.]]> Database schema design 101 for relational databases https://planetscale.com/blog/schema-design-101-relational-databases 2022-03-02T02:14:00.000Z 2022-03-02T02:14:00.000Z Camila Ramos Introducing Beam https://planetscale.com/blog/introducing-beam 2022-02-23T18:54:00.000Z 2022-02-23T18:54:00.000Z Jason Long Announcing Vitess 13 https://planetscale.com/blog/announcing-vitess-13 2022-02-22T19:10:00.000Z 2022-02-22T19:10:00.000Z Florent Poinsard How we made PlanetScale’s background jobs self-healing https://planetscale.com/blog/how-we-made-planetscale-background-jobs-self-healing-with-sidekiq 2022-02-17T15:05:34.027Z 2022-02-17T15:05:34.027Z Mike Coutermarsh Build a Laravel application with a MySQL database https://planetscale.com/blog/build-a-laravel-application-with-a-mysql-database 2022-02-15T16:10:00.000Z 2022-02-15T16:10:00.000Z Holly Guevara id(); $table->string('name'); $table->string('color'); }); } /** * Reverse the migrations. * * @return void */ public function down() { Schema::dropIfExists('moods'); } } Once ran, this will create the moods table with columns id, name, and color, as described below: id (UNSIGNED BIGINT) — Auto-increments to identify the mood name (VARCHAR) — Name of the mood color (VARCHAR) — Color used to represent the mood Next, open the migration file for entries at database/migrations/xxxx_xx_xx_xxxxxx_create_entries_table.php and replace it with:id(); $table->date('date'); $table->text('notes'); $table->foreignId('mood_id'); }); } /** * Reverse the migrations. * * @return void */ public function down() { Schema::dropIfExists('entries'); } } Here are the columns for the entries table: id (UNSIGNED BIGINT) — Auto-increments to identify the entry date (DATE) — The date of the entry notes (TEXT) — Any notes that go along with the entry mood_id (UNSIGNED BIGINT) — The corresponding mood for this entry While normally you could add the constrained() method to the foreign key (mood_id) to enforce referential integrity, we've purposely left it out here. PlanetScale previously did not support foreign key constraints enforced at the database level because we believe they aren't worth the trade-offs in performance and scalability. PlanetScale now supports foreign key constraints, so you can use the constrained() method if you would prefer to. Run migrations Now it’s time to run the migrations. In your terminal in the Laravel project directory, run the following:php artisan migrate Since you're connected to your PlanetScale database, these migrations are now live on your dev branch (or whatever you configured in your .env file)! To confirm this, go to your PlanetScale dashboard, click on your database, click "Branches", select the dev branch, click "Schema", and click "Refresh schema". You should see three tables: entries, migrations, and moods. You can also view your tables in the PlanetScale MySQL console by clicking "Console" and running:SHOW tables; DESCRIBE entries; DESCRIBE moods; Set up factories and seeders Let’s add some data to your database. Open up database/seeders/MoodSeeder.php and replace it with the following:insert([ 'name' => 'Happy', 'color' => '#FEC8DF', ]); DB::table('moods')->insert([ 'name' => 'Sad', 'color' => '#75CFE0', ]); DB::table('moods')->insert([ 'name' => 'Angry', 'color' => '#F5C691', ]); DB::table('moods')->insert([ 'name' => 'Productive', 'color' => '#C5E8B4', ]); DB::table('moods')->insert([ 'name' => 'Normal', 'color' => '#FFEFC9', ]); DB::table('moods')->insert([ 'name' => 'Calm', 'color' => '#BBA1D5', ]); } } As mentioned before, the moods table will be pretty static for now, so you can just explicitly create the data in the seeder since there isn’t much to it. For the entries seed data, you’ll want to generate several records with some random values instead of hard-coded data like in moods. This is where factories come into play. Open up database/factories/EntryFactory.php and replace it with: $this->faker->realText($maxNbChars = 300), 'mood_id' => Mood::inRandomOrder()->value('id'), ]; } } The text for each entry is being generated using Faker PHP. A random id from the Mood model is assigned for mood_id. The date entry is a little more complicated because it needs to be unique and in a specific format. You can’t use the Faker library for this because it can only generate a unique DATETIME value, not DATE. You’ll create the random date values in the next step. Finally, modify your main database/seeders/DatabaseSeeder.php file as follows:call(MoodSeeder::class); // create an array of random unique dates in the format y-m-d $randomDates = []; while (count($randomDates) < 15) { $date = Carbon::today()->subDays(rand(0, 31))->format('Y-m-d'); if (!in_array($date, $randomDates)) array_push($randomDates, $date); } foreach($randomDates as $date) { Entry::factory()->create([ 'date' => $date ]); } } } This first runs the MoodSeeder.php file that you filled out earlier. Next, you're creating an array of 15 random, unique dates using the Carbon library. Finally, you loop through that array, call the database/factories/EntryFactory.php file that you created in the previous step, and add the random date to each entries record. The EntryFactory uses the create() method to create new database records based on the Entry model. Set up models The final step before seeding is to set up the models. First, open up app/Models/Mood.php and replace it with:hasMany(Mood::class); } // Clear moods cache upon modifying a mood entry protected static function boot() { parent::boot(); static::saving(function() { Cache::forget('moods'); }); } } Here, you're first specifying what attributes can be modified in fillable. Forgetting to set this is a common mistake that can be difficult to debug as a beginner, so any time you add a column that you may write to, make sure to update it here! You're also setting timestamps to FALSE so that Laravel doesn’t automatically create created_at and updated_at columns in the moods table. Models also allow you to define Eloquent relationships. In the Mood model, you're defining the one-to-many relationship between entries and moods. Each mood can have several entries, but each entry will only have one mood that corresponds to it. This is reflected in the entries() function using the hasMany() method. Each Mood has many entries. The boot() method is used to clear the cache upon saving a new mood. You’ll see where this comes into play when you update your controllers. Next, open app/Models/Entry.php and replace it with the following:belongsTo(Mood::class); } // Clear entries cache upon modifying an entry protected static function boot() { parent::boot(); static::saving(function() { Cache::forget('entries'); }); } } This is similar to the Mood model. You're also creating the inverse relationship using belongsTo(). Each Entry belongs to exactly one Mood. Defining these relationships now will allow you to use Eloquent, Laravel's ORM, to easily work with your data. Seed your database Finally, it’s time to seed your database! In the terminal in your project folder, run the following:php artisan db:seed This will run the main database/seeders/DatabaseSeeder.php file. To view your seeded data and confirm that it worked, head back to your PlanetScale dashboard, select the database, click "Branches", select the dev branch, and click "Console". Run the following queries:SELECT * FROM moods; You should see the data for the moods table you created.SELECT * FROM entries; For the entries table, you have: 15 records with random, unique dates from the past 2 months, random text under notes, and a randomly selected mood_id that matches one of the moods in the moods table. Now that your development branch is loaded up with some mock data, let’s set up the resource controllers so you can create and modify the data. Add controllers While the database is set up and ready to go, the application doesn’t actually do anything yet. Let’s fix that! EntryController First, open app/Http/Controllers/EntryController.php and replace it with:get(); }); return view('entries.index') ->with('entries', $entries); } /** * Show the form for creating a new resource. * * @return \Illuminate\Http\Response */ public function create() { $moods = Cache::remember('moods', 3600, function() { return Mood::all(); }); return view('entries.create') ->with('moods', $moods); } /** * Store a newly created resource in storage. * * @param \Illuminate\Http\Request $request * @return \Illuminate\Http\Response */ public function store(Request $request) { // Date must be in format Y-M-D. Must also not already exist in entries table date column // Selected mood_id must exist in the moods table under column id $request->validate([ 'date' => 'required|date_format:Y-m-d|unique:entries,date,', 'notes' => 'string|nullable', 'mood_id' => 'required|exists:moods,id', ]); Entry::create($request->all()); return redirect()->route('entries.index') ->with('success', 'Entry created.'); } /** * Display the specified resource. * * @param \App\Models\Entry $entry * @return \Illuminate\Http\Response */ public function show(Entry $entry) { return view('entries.show') ->with('entry', $entry); } /** * Show the form for editing the specified resource. * * @param \App\Models\Entry $entry * @return \Illuminate\Http\Response */ public function edit(Entry $entry) { $moods = Cache::remember('moods', 3600, function() { return Mood::all(); }); return view('entries.edit', compact('entry', 'moods')); } /** * Update the specified resource in storage. * * @param \Illuminate\Http\Request $request * @param \App\Models\Entry $entry * @return \Illuminate\Http\Response */ public function update(Request $request, Entry $entry) { // Date must be in format Y-M-D. Must also not already exist in entries table date column // Selected mood_id must exist in the moods table under column id $request->validate([ 'date' => 'required|date_format:Y-m-d|unique:entries,date,'. $entry->id, 'notes' => 'string|nullable', 'mood_id' => 'required|exists:moods,id', ]); $updatedEntry = $request->all(); $entry->update($updatedEntry); return redirect()->route('entries.show', [$entry->id]) ->with('success', 'Entry updated.'); } /** * Remove the specified resource from storage. * * @param \App\Models\Entry $entry * @return \Illuminate\Http\Response */ public function destroy(Entry $entry) { $entry->delete(); return redirect()->route('entries.index') ->with('success', 'Entry deleted.'); } } Then EntryController has the following methods: index() — Display all entries create() — Display the form to create a new entry store() — Validate new entry input and save it to the database show() — Display a single entry edit() — Display the form to update an entry update() — Validate updated entry input and update in the database destroy() — Delete an entry Let’s go over a few notable details of this controller that you’ll also see in the MoodController. Caching The index() method grabs ALL entries from the database. While this isn’t a huge number for this sample application, it could potentially turn into a huge performance and cost hit as the data grows. There are a few ways to improve the performance, but one quick solution is to cache the data.$entries = Cache::remember('entries', 3600, function() { return Entry::orderBy('date', 'ASC')->get(); }); Laravel makes caching easy using the Cache::remember() method. This will check the cache to see if the data already exists there, and if not, it will pull from the database and store it in the cache for 3600 seconds as entries. For a more in-depth primer on Laravel caching, check out Introduction to Laravel caching. Views With every method, you’ll see a return statement at the end that either returns a view or a redirect along with some data.return view('entries.index') ->with('entries', $entries); In the above example, after the method executes, the user will be routed to the entries index page found at resources/views/entries/index.php (you’ll create this soon). The data for $entries will also be passed to the view. Form validation The store() and update() methods both require some kind of form validation before storing the entries to the database. You should never trust user input, so backend form validation is essential for, well, validating that the user's input is correct.$request->validate([ 'date' => 'required|date_format:Y-m-d|unique:entries,date,'. $entry->id, 'notes' => 'string|nullable', 'mood_id' => 'required|exists:moods,id', ]); Laravel makes even the most complex form validation a breeze. The above code snippet is from the update() method. Let’s examine the date validation. The first two, required and date_format, are pretty straightforward. The next one, unique:table,column, is a little more complex. You don’t want repeated dates in this application, so you must check that the date is unique when validating. However, if you're updating an existing entry, your application will compare the user's updated input to the existing entry. If the user is only updating the text, then the date will be the same, so it will fail validation. To get around this, you can pass in the current id and it will check that all dates are unique except for the date on the specified entry. HomeController Next, set up the HomeController, which will be used to grab the data for the homepage. Open up app/Http/Controller/HomeController.php and paste in the following:get(); }); $moods = Cache::remember('moods', 3600, function() { return Mood::all(); }); return view('home', compact('entries', 'moods')); } } MoodController Finally, open up app/Http/Controllers/MoodController.php and paste in the following:with('moods', $moods); } /** * Show the form for creating a new resource. * * @return \Illuminate\Http\Response */ public function create() { return view('moods.create'); } /** * Store a newly created resource in storage. * * @param \Illuminate\Http\Request $request * @return \Illuminate\Http\Response */ public function store(Request $request) { $this->validate($request, [ 'name' => 'required|string', 'color' => 'required|string', ]); Mood::create($request->all()); return redirect()->route('entries.mood') ->with('success', 'Mood created.'); } /** * Display the specified resource. * * @param \App\Models\Mood $mood * @return \Illuminate\Http\Response */ public function show(Mood $mood) { return view('moods.show') ->with('mood', $mood); } /** * Show the form for editing the specified resource. * * @param \App\Models\Mood $mood * @return \Illuminate\Http\Response */ public function edit(Mood $mood) { return view('moods.edit')->with('mood', $mood); } /** * Update the specified resource in storage. * * @param \Illuminate\Http\Request $request * @param \App\Models\Mood $mood * @return \Illuminate\Http\Response */ public function update(Request $request, Mood $mood) { $this->validate($request, [ 'name' => 'required|string', 'color' => 'required|string', ]); $updatedMood = $request->all(); $mood->update($updatedMood); return redirect()->route('moods.show', [$mood->id]) ->with('success', 'Mood updated.'); } /** * Remove the specified resource from storage. * * @param \App\Models\Mood $mood * @return \Illuminate\Http\Response */ public function destroy(Mood $mood) { $mood->delete(); return redirect()->route('moods.index') ->with('success', 'Mood deleted.'); } } This is very similar to the EntryController, so for the sake of brevity, I won’t expand on any of the details. Set up routes Now that you have controllers created, let’s set up the routes. Open up routes/web.php and replace it with: @yield('title') - Mood Tracker
@yield('content')
home.blade.php Next, open up resources/views/home.blade.php and paste in the following:@section('title', 'Home') @extends('layout') @section('content')
@foreach($entries as $entry)
{{ \Carbon\Carbon::parse($entry->date)->format('m/d') }}
@endforeach
Legend
    @foreach($moods as $mood)
  • {{ $mood->name }}
    • @endforeach
@endsection This file uses the HomeController to render the homepage. If you look at that controller, you’ll see the moods and entries data are both retrieved from the database and passed to this view. You're then looping through both to display on the page. entries/index.blade.php This file, along with the next three files, will render the views for the following pages: Landing page that shows all entries Page to view a single entry Page to edit an entry Page to create a new entry Open up entries/index.blade.php and paste in the following:@section('title', 'Entries') @extends('layout') @section('content')
@foreach($entries as $entry) @endforeach
Date Mood Notes Edit Delete
{{ \Carbon\Carbon::parse($entry->date)->format('M d, Y') }}
{{ $entry->mood->name }}

{{ $entry->notes }}

Edit
@csrf @method('DELETE')
@endsection This loops through all entries and displays them. It also includes the button to get to the individual edit page and a button for deletion. entries/edit.blade.php Open up entries/edit.blade.php and paste in:@section('title', 'Edit entry') @extends('layout') @section('content')

Edit your mood entry

How are you feeling?

@csrf @method('PATCH')
Date must be in format YYYY-MM-DD
@if ($errors->any()) @endif
@error('title')
{{ $message }}
@enderror @endsection Since the resource controller uses PATCH, you're using a hidden PATCH method on the form with @method('PATCH'). You're also validating this input on the backend in EntryController.php, so if the input is invalid, the error messages are displayed here with {{ $error }}. Another thing to note is this view needs to show the existing entry that’s being updated, so you're setting the value for each input with the entry data that’s passed in from the database. entries/create.blade.php Next, update the view used to create a new entry. Open up entries/create.blade.php and paste in the following:@section('title', 'New entry') @extends('layout') @section('content')

Create a new entry

How are you feeling today?

@csrf
@if ($errors->any()) @endif
@endsection This is similar to the edit view, but without any existing data being pulled in. You can even get crafty and consolidate the two views, but I personally prefer to keep them separate so that it’s easier to read. entries/show.blade.php Finally, create the view that’s used to display a single entry. Open up entries/show.blade.php and paste in:@section('title', 'Entry') @extends('layout') @section('content')

Entry

Edit
Date
{{ \Carbon\Carbon::parse($entry->date)->format('M d, Y') }}
Mood
{{ $entry->mood->name }}
Notes
{{ $entry->notes }}
@endsection Mood views The views for creating, updating, and displaying the moods are almost identical to those for entries, so you copy them in straight from the final repo. You can find the code for them in this section on the GitHub repository. Add, update, and delete data Your application is now complete and ready to play with! Let’s test it out. Make sure you still have everything running: Start the PHP server:php artisan serve Run the build process for Tailwind:npm run watch Navigate to http://localhost:8000 to view your app. Add an entry Click on the "New entry" button at the top right and fill out the form. Try to choose a date that’s already been taken or leave the required mood field blank and you’ll get a validation error, as expected. Once you submit a valid entry, you’ll be taken back to the entries index page where you’ll see the entry listed. You can also click the "Edit" button to modify an entry, or the "Delete" button to get rid of one. This sample app doesn’t have a delete confirmation built in, so don’t click unless you're sure you want to delete it! Deploy development database branch to production PlanetScale offers branching capabilities, similar to the Git model. When you started working on this application, you created a development branch off of the main production branch. You’ve spent this whole time working in that development database branch, making schema changes as needed. But that production database branch is still empty. So the next step is to merge this development branch into production. You do this by opening a PlanetScale deploy request (similar to a GitHub pull request). You can view your schema diff here and PlanetScale will check to make sure there are no merge conflicts. Once everything is good, you can deploy the changes straight to production with zero downtime. It’s really that simple! Let’s create a deploy request and merge this dev branch into production. Create a deploy request In your PlanetScale dashboard, select the database, click "Branches", and select the dev branch. On the Overview page, you’ll see the Deploy Request form. Make sure "Deploy to" is set to main. Write a comment to go with your deploy request, and then click "Create deploy request". Once created, you’ll see a schema diff under "Schema changes" that shows you exactly what changes this deploy request will introduce if merged. This allows you and/or your team to carefully review schema changes before pushing them to production. Deploy schema changes to production Once the changes are approved, it’s time to merge the deploy request. Click "Add changes to the deploy queue". As the changes are deploying, you’ll see the deployment progress for each table. One really cool feature to note here is that these schema changes are being updated with zero downtime or locking. The branching feature allows PlanetScale to offer non-blocking schema changes, so your production application will continue to work seamlessly as these changes are deployed. PlanetScale is handling it all in the background. You can now go to your main branch, click "Schema", and you’ll see the schema you just created in development now live in production! Recap If you’ve reached the end, congratulations! You should have a working mood tracker application complete with all CRUD functionality and a production MySQL database. Throughout the tutorial, you learned how to: Create Laravel 9 controllers, models, migrations, factories, and seeders Create Laravel forms Validate input from Laravel forms Work with Laravel Eloquent ORM Connect your Laravel application to a MySQL database Create PlanetScale deploy requests Please let me know if you have any questions! You can find me on Twitter at @hollylawly. Thanks for reading!]]> How to seed a database with Prisma and Next.js https://planetscale.com/blog/how-to-seed-a-database-with-prisma-and-next-js 2022-02-11T17:11:00.000Z 2022-02-11T17:11:00.000Z James Q Quick :@/?sslaccept=strict In this starter code, we have two different models configured, Product and Category in the schema.prisma file.model Product { id Int @id @default(autoincrement()) name String description String price Decimal image String category Category? @relation(fields: [category_id], references: [id]) category_id Int } model Category { id Int @id @default(autoincrement()) name String description String products Product[] } Before you can run a seed script, you’ll need to push this schema to your database.npx prisma db push Create the seed script The prisma directory is a convenient place to include a seed script since this is where the schema.prisma file referenced above is located. Inside of this directory of the starter code, you’ll see a seed.js file. Notice also the data.js file which exports sample data that you will use when the seed script is run. Although the seed script is finished in the starter repository, let’s break down the steps of how you would create it yourself from scratch. First, you’ll need to import the PrismaClient and the categories and products data. Then, you’ll need to generate a new client by calling PrismaClient().const { PrismaClient } = require('@prisma/client') const { categories, products } = require('./data.js') const prisma = new PrismaClient() You will need to use the CommonJS syntax for imports and exports in your JavaScript files. This is different from the ECMAScript modules syntax you're used to using inside of a Next.js project. This is because this file is being run on its own, outside of the running Next.js application. After you’ve got your imports, create a load() function. This where the actual database seeding will take place. Make sure to mark the function as async since you will use the await keyword inside of it. Also, don’t forget about error handling. Go ahead and add a try/catch/finally block inside of your function to handle errors and disconnect from your database after the seeding has completed.const load = async () => { try { } catch (e) { console.error(e) process.exit(1) } finally { await prisma.$disconnect() } } load() With the load() function set up, you can start to add data to your database by passing the categories and products arrays to the appropriate createMany() function.await prisma.category.createMany({ data: categories }) console.log('Added category data') await prisma.product.createMany({ data: products }) console.log('Added product data') Your script should now be set up to add data, but one thing you’ll want to do first is delete any existing data. This way, you can verify that your database will be populated in exactly the same way each time it is seeded. Before the lines you just added for creating data, call deleteMany() for both tables.await prisma.category.deleteMany() console.log('Deleted records in category table') await prisma.product.deleteMany() console.log('Deleted records in product table') Lastly, the dummy data maintains a relationship between an individual product and its corresponding category with the category_id property. Because of this, this category_id property is prepopulated with the product records. However, since the id properties of products and categories are auto-incremented, you’ll need to manually reset them to 0. This will ensure that each category_id will correspond to the appropriate category record. You can reset the auto-incremented values by calling the prisma.$queryRaw function and passing the appropriate SQL statement like so.await prisma.$queryRaw`ALTER TABLE Product AUTO_INCREMENT = 1` console.log('reset product auto increment to 1') await prisma.$queryRaw`ALTER TABLE Category AUTO_INCREMENT = 1` console.log('reset category auto increment to 1') Here’s what the full file looks like.const { PrismaClient } = require('@prisma/client') const { categories, products } = require('./data.js') const prisma = new PrismaClient() const load = async () => { try { await prisma.category.deleteMany() console.log('Deleted records in category table') await prisma.product.deleteMany() console.log('Deleted records in product table') await prisma.$queryRaw`ALTER TABLE Product AUTO_INCREMENT = 1` console.log('reset product auto increment to 1') await prisma.$queryRaw`ALTER TABLE Category AUTO_INCREMENT = 1` console.log('reset category auto increment to 1') await prisma.category.createMany({ data: categories }) console.log('Added category data') await prisma.product.createMany({ data: products }) console.log('Added product data') } catch (e) { console.error(e) process.exit(1) } finally { await prisma.$disconnect() } } load() Configure the seed command There are a couple of different ways to configure your seed script to run. Add a new script in the package.json The first option is to define your own script inside of the package.json. Inside of the scripts section add the following line."seed": "node prisma/seed.js"` This will enable you to run npm run seed to run your seed script. Go ahead and give it a try! You should see success log messages in your console. Add a prisma.seed field in package.json The second way to configure your seed script is to tap into the Prisma configuration in your package.json. For this to work you can add the following line at the top level of your package.json"prisma": { "seed": "node prisma/seed.js" }, With that configuration added, you can now trigger your seed script by running npx prisma db seed. Give that a shot! So far, this is a pretty similar result to what we had in the previous step. However, there is a bit more happening behind the scenes. Because the prisma.seed property is defined, Prisma will automatically run the seed command when either or the following commands are run: npx prisma migrate dev or prisma migrate reset. Whether or not you want this to happen is totally up to you. Personally, I prefer to choose when the seeding should take place, so I would prefer the first option by configuring it in the scripts section. Wrap up Hopefully, this tutorial gave you a good understanding of how to automically populate your PlanetScale database by configuring a seed script with Prisma. If you have any additional questions, let us know on Twitter.]]> Defining the database maturity model https://planetscale.com/blog/defining-the-database-maturity-model 2022-02-10T13:00:00.000Z 2022-02-10T13:00:00.000Z Nick Van Wiggeren Introduction to Laravel caching https://planetscale.com/blog/introduction-to-laravel-caching 2022-02-09T16:30:00.000Z 2022-02-09T16:30:00.000Z Holly Guevara "Create new database". Give your database a name and select the region closest to you. Select a cluster size and storage size. Enter your payment information, then click "Create database". Once it’s finished initializing, you’ll land on the Overview page for your database. Click on the "Branches" tab and select the main branch. This is a development branch that you can use to modify your schema. PlanetScale has a database workflow similar to the Git branching model. While developing, you can: Create new branches off of your main branch Modify your schema as needed Create a deploy request (similar to a pull request) Merge the deploy request into main Leave this page open, as you’ll need to reference it soon. Set up Laravel app Next, let’s set up the pre-built Laravel 9 application. This comes with a simple CRUD API that displays random bogus sentences (we’re going to think of them as robot quotes) along with the quote author's name. The data for both of these columns are auto-generated using Faker. There is currently no caching in the project, so you’ll use this starter app to build on throughout the article. For this tutorial, you’ll use the default file-based cache driver, meaning the cached data will be stored in your application's file system. This is fine for this small application, but for a bigger production app, you may want to use a different driver. Fortunately, Laravel supports some popular ones, such as Redis and Memcached. Before you begin, make sure you have PHP (this article is tested with v8.1) and Composer (at least v2.2) installed. Clone the sample application:git clone -b starter https://github.com/planetscale/laravel-caching Install the dependencies:composer install Copy the .env.example file to .env:mv .env.example .env Next, you need to connect to your PlanetScale database. Open up the .env file and find the database section. It should look like this:DB_CONNECTION=mysql DB_HOST= DB_PORT=3306 DB_DATABASE= DB_USERNAME= DB_PASSWORD= MYSQL_ATTR_SSL_CA=/etc/ssl/cert.pem For DB_DATABASE, you can use your PlanetScale database name directly if you have a single unsharded keyspace. If you have a sharded keyspace, you'll need to use @primary. This will automatically direct incoming queries to the correct keyspace/shard. For more information, see the Targeting the correct keyspace documentation. Go back to your PlanetScale dashboard to the main branch page for your database. Click "Connect" in the top right corner. Click "Generate new password". Select "Laravel" from the dropdown (it’s currently set to "General"). Copy this and replace the .env content highlighted in Step 4 with this connection information. It’ll look something like this:DB_CONNECTION=mysql DB_HOST=xxxxxxxx.xx-xxxx-x.psdb.cloud DB_PORT=3306 DB_DATABASE=xxxxxxxx DB_USERNAME=xxxxxxxxxxxxx DB_PASSWORD=pscale_pw_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx MYSQL_ATTR_SSL_CA=/etc/ssl/cert.pem Make sure you save the password before leaving the page, as you won’t be able to see it again. Your value for MYSQL_ATTR_SSL_CA may differ depending on your system. Run the migrations and seeder:php artisan migrate php artisan db:seed Start your application:php artisan serve You can now view your non-cached data from the PlanetScale database in the browser at http://localhost:8000/api/quotes. Project structure overview Before diving in, let’s explore the project and relevant files. Quote controller The sample application has a single controller, app/Http/Controllers/QuoteController.php, that has methods to display, update, create, and delete quotes. Since this is an API resource controller, you don’t need to include the usual show and edit controllers that only return views. You’ll take a closer look at this soon once you add caching, but right now, nothing is cached. Quote model There’s also a model, app/Models/Quote.php, where you can define how Eloquent interacts with the quotes table. Since you only have one table in this application, there are no relationships or interactions, so the model is pretty barebones right now:class Quote extends Model { use HasFactory; public $timestamps = FALSE; protected $fillable = [ 'text', 'name' ]; } You’ll revisit it soon, though, once you implement caching. Quote migration Next up is the initial quotes migration file, database/migrations/2022_01_215158_create_quotes_table.php. When you ran the migrations in the previous step, this file created the quotes table with the specified schema:public function up() { Schema::create('quotes', function (Blueprint $table) { $table->id(); $table->text('text'); $table->string('name'); }); } Quote factory and seeder Finally, there’s a factory and seeder. The factory, database/factories/QuoteFactory.php, uses Faker to create mock sentence and author data. The seeder, database/seeders/DatabaseSeeder.php, then runs this factory 100 times to create 100 rows of this Faker-generated data in the quotes table.public function definition() { return [ 'text' => $this->faker->realText(100, 3), 'name' => $this->faker->name() ]; } When you ran php artisan migrate and php artisan db:seed in the previous steps, these are the files that were ran. Queries without caching Before you add caching, it’s important to see how the application currently perform so you know the effect that caching has. And how will you do that if you don’t know what your query performance was before adding caching? Let’s run some queries and see how long they take to complete. Get all data Open up the app/Http/Controllers/QuoteController.php file and go to the index() method. Replace it with:public function index() { $startTime = microtime(true); // start timer $quotes = Quote::all(); // run query $totalTime = microtime(true) - $startTime; // end timer return response()->json([ 'totalTime' => $totalTime, 'quotes' => $quotes ]); } The PHP function, microtime(true), provides an easy way to track the time before and after the query. You can also use an API testing tool like Postman to see the time it takes to complete. Let’s call the API endpoint to check how long it currently takes to pull all of this data from the database. Open or refresh http://localhost:8000/api/quotes in your browser. You’ll now see a totalTime value that displays the total time in seconds that it took to execute this query. The total time will fluctuate, but I’m personally getting anywhere between 0.9 seconds and 2.3 seconds! Of course, you'd want to paginate or chunk your data in most cases, so hopefully, it wouldn’t take several seconds to grab in the first place. But caching can still greatly reduce the time it takes to get data from this endpoint after the initial hit. Let’s add caching now. Add caching to your Laravel app Open up app/Http/Controllers/QuoteController.php, bring in the Cache facade at the top of the file, and replace the $quotes = Quote::all(); in index() with:// ... use Illuminate\Support\Facades\Cache; // ... public function index() { // ... $quotes = Cache::remember('allQuotes', 3600, function() { return Quote::all(); }); // ... } // ... Now let’s hit that API endpoint again. Refresh the page at http://localhost:8000/api/quotes. If this is your first time running the call, you’ll have to refresh again for the caching to take effect. Check out the new time I’m getting: 0.0006330013275146484 seconds! Before caching, this exact same query took between 0.9 seconds and 2.3 seconds. Incredible, right? Even though this seems to be a massive improvement on the surface, there are still some issues that you need to tackle. Let’s first dissect the Cache::remember() method and then go over some gotchas with this addition. If at any time you need to clear the cache manually while testing, you can run the following in your terminal:php artisan cache:clear Caching with remember() The Cache::remember() method first tries to retrieve a value from the cache. If that value doesn’t exist, it will go to the database to grab the value, and then store it in the cache for future lookups. You will specify the name of the value and how long it stores it, as shown below:$quotes = Cache::remember('cache_item_name', $timeStoredInSeconds, function () { return DB::table('quotes')->get(); }); This method is super handy because it does several things at once: checks if the item exists in cache, grabs the data if not, and stores it in the cache once grabbed. If you prefer just to grab the value from cache and do nothing if it doesn’t exist, use:$value = Cache::get('key'); If you want to grab the value from cache and pull it from the database if it doesn’t exist, use:$value = Cache::get('key', function () { return DB::table(...)->get(); }); This one is similar to remember(), except it doesn’t store it in the cache. Inconsistent data in the cache So what are the problems that you need to deal with? Let’s see one of them in action. Refresh the [http://localhost:8000/api/quotes](http://localhost:8000/api/quotes) page in the browser one more time to make sure the cache hasn’t expired. Now, add a new record to the quotes table by pasting the following in your terminal:curl -X POST -H 'Content-Type: application/json' -d '{ "text": "If debugging is the process of removing software bugs, then programming must be the process of putting them in.", "name": "Edsger Dijkstra" }' http://localhost:8000/api/quotes -i You should get a HTTP/1.1 200 OK response along with the newly added record. Now go back to your Quotes page in the browser and refresh. The new data you added isn’t there! That’s because you just wrote this item to the database, but you're not actually going to the database to retrieve it. The cache has no idea it exists. You can confirm it was added to the database by going back to your PlanetScale dashboard, select the database, click "Branches", and click "Console". Run the following command and you should see 101 records:select * from quotes; Scroll to the bottom and you’ll see the newly added quote. You can also query it directly by id:select * from quotes where id=101; Solving the write problem If it’s important for your application to always show the most up-to-date data, one quick way to fix this is using the Quote model's booted() method. Open up app/Models/Quote.php and replace it with: Using the PlanetScale CLI with GitHub Actions workflows https://planetscale.com/blog/using-the-planetscale-cli-with-github-actions-workflows 2022-02-03T16:11:19.285Z 2022-02-03T16:11:19.285Z Taylor Barnett /demo-db. DB_NAME: Your database name in PlanetScale You are ready to run the action! Push (or create) a branch in your GitHub repository that starts with db/. For example, I want to develop a new feature that will require database changes. I would name my branch db/new-feature, and my PlanetScale branch will be called new-feature. Go back to your repo and under "Actions" you’ll see the new workflow running! Extra credit! (This is an optional step.) Once you have tried automatically creating database branches, what if you wanted to open a deploy request in PlanetScale when the branch is created, so it is ready for when you want to merge a database change? If you look in ps-create-helper-functions.sh you can find this function:function create-deploy-request { local DB_NAME=$1 local BRANCH_NAME=$2 local ORG_NAME=$3 local raw_output=`pscale deploy-request create "$DB_NAME" "$BRANCH_NAME" --org "$ORG_NAME" --format json` if [ $? -ne 0 ]; then echo "Deploy request could not be created: $raw_output" exit 1 fi local deploy_request_number=`echo $raw_output | jq -r '.number'` # if deploy request number is empty, then error if [ -z "$deploy_request_number" ]; then echo "Could not retrieve deploy request number: $raw_output" exit 1 fi local deploy_request="https://app.planetscale.com/${ORG_NAME}/${DB_NAME}/deploy-requests/${deploy_request_number}" echo "Check out the deploy request created at $deploy_request" # if CI variable is set, export the deploy request URL if [ -n "$CI" ]; then echo "::set-output name=DEPLOY_REQUEST_URL::$deploy_request" echo "::set-output name=DEPLOY_REQUEST_NUMBER::$deploy_request_number" create-diff-for-ci "$DB_NAME" "$ORG_NAME" "$deploy_request_number" "$BRANCH_NAME" fi } This function will create a deploy request in PlanetScale and then export the deploy request URL and deploy request number to the GitHub Action output. If you are running this action in a CI environment, it will also create a diff for the deploy request. You can add this to your .pscale/cli-helper-scripts/create-branch.sh at the end like this:create-deploy-request "$DB_NAME" "$BRANCH_NAME" "$ORG_NAME" If you don’t have your main branch in PlanetScale promoted to production, you need to do this before rerunning the workflow. And then push (or create) a new db/** branch in GitHub to rerun this. What GitHub Action workflows would you like to see? We want to hear from you! Your ideas might appear in a future blog post or example! Now that you have an idea of what it is like to automate workflows with GitHub Actions and the PlanetScale CLI, what are some workflows you would like to see built? What manual steps do you do with your databases that you wish were automated while benefiting from branching, deploy requests, and non-blocking schema changes? What workflows do you want to see based on triggers in GitHub issues and pull requests? We would love to hear your feedback! Tweet at us @planetscale to tell us what you would like to build or see with PlanetScale and automated workflows.]]> Using entropy for user-friendly strong passwords https://planetscale.com/blog/using-entropy-for-user-friendly-strong-passwords 2022-01-24T16:57:52.732Z 2022-01-24T16:57:52.732Z Mike Coutermarsh " src="/users/password-strength" required="">
<%= f.label(:password, "New password", class: "mb-0") %>
<%= f.password_field(:password, class: "js-password-strength", autofocus: true, autocomplete: "new-password", required: true) %> This form element also has a tiny bit of extra JavaScript added to update the meter after each check.<% # Make 10% the min so _some_ red appears. # For values > 90, keep arc slight unclosed. strength = 10 if strength < 10 strength = 90 if (strength > 90) && (strength < 100) radius = 40 perimeter = Math::PI * radius * 2 stroke_dashoffset = (perimeter - (perimeter * strength)) / 100 arc_color = case strength when 0..33 "rgba(var(--red-500))" when 33..66 "var(--orange-500)" when 66..100 "var(--yellow-500)" end %> <%= strength == 100 ? "Strong" : "Too weak" %> <% if strength == 100 %> <% else %> <% end %> Password strength calculation The controller code determines the password strength percentage and then renders the meter.def create checker = User.password_checker entropy = checker.calculate_entropy(params[:value] || "") percentage = (entropy / STRONG_ENTROPY) * 100 percentage = 100 if percentage > 100 render(partial: "users/shared/password_strength_meter", locals: { strength: percentage.to_i }) end Learn more on how to implement entropy-based password forms Give it a try yourself by playing around with our sign up form: https://auth.planetscale.com/sign-up We found these resources useful when implementing our password strength meter: How to calculate password strength Strong_password gem Password strength test tool]]> How to set up Next.js with Prisma and PlanetScale https://planetscale.com/blog/how-to-setup-next-js-with-prisma-and-planetscale 2022-01-20T20:19:00.000Z 2022-01-20T20:19:00.000Z Camila Ramos { e.preventDefault() const body = { firstName, email, subject, message } try { const response = await fetch('/api/inquiry', { method: 'POST', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify(body) }) if (response.status !== 200) { console.log('something went wrong') //set an error banner here } else { resetForm() console.log('form submitted successfully !!!') //set a success banner here } //check response, if success is false, dont take them to success page } catch (error) { console.log('there was an error submitting', error) } } const resetForm = () => { setFirstName('') setEmail('') setSubject('') setMessage('') } In this case, I am using the useState hook and setting the state for each variable (firstName, email, subject, and message) by passing an anonymous function to the onChange property of the form inputs. setFirstName(e.target.value)} value={firstName} className='bg-zinc-300 text-gray-200-900 focus:ring-indigo-400 focus:border-indigo-400 border-warm-gray-300 block w-full rounded-md px-4 py-3 shadow-sm' /> You’ll have to call the handleSubmit function somewhere to execute this code. Because I’m using a form, I can pass the function call to the onSubmit property like this:
handleSubmit(e)} All done! Now you’re ready to deploy your database to work in production and take your app live. Deploying to production Navigate back to your PlanetScale database. Hit the connect button and select Prisma from the dropdown menu. Hit the button to generate a new password, and be sure to copy/paste this somewhere for you to access later. In my example, I’m using Netlify. You can use Vercel to deploy your app, and the steps will be similar. In my case, the project was already deployed, so I’ll have to go back and make some changes to my environment variables and redeploy. If you are deploying this project for the first time, you can set the environment variables in your initial configuration and won’t have to redeploy as outlined below. Create a Netlify account and connect the GitHub repo that is connected to this project. Navigate to Site Settings. Using the side navigation, go to Build and deploy, and select Environment. Add a variable called DATABASE_URL and set the value to be the URL you were given from your PlanetScale-generated password. Be sure to remove the quotes that wrap the URL. Save these changes. In the Deploys tab, hit the button that says Trigger Redeploy. Now you’re ready to either push this code to your main branch or merge the branch you’re working on into main to see your new database live. Give yourself a pat on the back because you just deployed your first PlanetScale database 🥳. Try it out Follow this guide and spin up a working app in just a few minutes! Create a new database, define your data models, and write your API to write to your database directly from your Next app. Tweet the team with any questions you have @planetscale.]]> How our Rails test suite runs in 1 minute on Buildkite https://planetscale.com/blog/how-our-rails-test-suite-runs-in-1-minute-on-buildkite 2022-01-18T15:37:51.755Z 2022-01-18T15:37:51.755Z Mike Coutermarsh Introducing Prisma’s Data Platform PlanetScale integration https://planetscale.com/blog/planetscale-mysql-database-on-prisma-platform 2021-11-18T14:55:55.366Z 2021-11-18T14:55:55.366Z Taylor Barnett Bring your data to PlanetScale https://planetscale.com/blog/import-your-mysql-data-to-planetscale 2021-11-17T15:30:00.000Z 2021-11-17T15:30:00.000Z Phani Raju PlanetScale is GA https://planetscale.com/blog/ga 2021-11-16T00:03:57.138Z 2021-11-16T00:03:57.138Z Sam Lambert Introducing PlanetScale Managed on AWS and GCP https://planetscale.com/blog/introducing-planetscale-managed 2021-11-03T15:00:00.000Z 2023-10-18T15:00:00.000Z James Cunningham New PlanetScale pricing: Scaler plan upgrades and our new enterprise plan https://planetscale.com/blog/introducing-new-planetscale-pricing 2021-10-28T15:40:00.000Z 2021-10-28T15:40:00.000Z Sam Lambert Comparing AWS’s RDS and PlanetScale https://planetscale.com/blog/planetscale-vs-aws-rds 2021-09-30T15:52:00.000Z 2021-09-30T15:52:00.000Z Jarod Reyes Quick deploys using the Web Console https://planetscale.com/blog/sql-in-web-console 2021-09-13T15:18:00.000Z 2021-09-13T15:18:00.000Z Elom Gomez Optimizing SQL with Query Statistics https://planetscale.com/blog/optimizing-sql-with-query-statistics 2021-08-31T15:12:41.189Z 2021-08-31T15:12:41.189Z David Graham explain select * from backup where deleted_at is null and expires_at <= current_timestamp order by id asc limit 1000\G; *************************** 1. row *************************** id: 1 select_type: SIMPLE table: backup partitions: NULL type: ALL possible_keys: index_backup_on_expires_at_and_data_deleted_at_and_deleted_at key: NULL key_len: NULL ref: NULL rows: <100s of thousands—every row in the table> filtered: 5.00 Extra: Using where A couple observations about the data reveals why this is a poor index and points us to a more selective index that will narrow this result set dramatically. 97% of the backup rows are expired, so querying by expires_at < current_timestamp is not selective. 97.5% of the rows are soft deleted, so querying by deleted_at is not null is highly selective. In retrospect, this makes sense when dealing with daily backups that replace the previous day's backup. However, most of our tables do not match this workload and the relationship is the reverse: most rows are live rather than soft deleted. We often include deleted_at as a trailing key in composite indexes for this reason, but this isn’t quite right for the backup table. If we change the order of keys in this index to (deleted_at, expires_at, data_deleted_at) then the query is highly selective. It can eliminate deleted rows from the set and search over the few remaining rows that may be expired. The explain plan for the new index shows that the query planner does indeed choose the index and estimates it needs to visit only a single row to provide the result.> explain select * from backup where deleted_at is null and expires_at <= current_timestamp order by id asc limit 1000\G; *************************** 1. row *************************** id: 1 select_type: SIMPLE table: backup partitions: NULL type: range possible_keys: index_backup_on_deleted_at_and_expires_at_and_data_deleted_at key: index_backup_on_deleted_at_and_expires_at_and_data_deleted_at key_len: 17 ref: NULL rows: 1 filtered: 100.00 Extra: Using index condition; Using filesort Conclusion Using PlanetScale query statistics along with MySQL explain plans to optimize an index reduced a 719ms query to under 20ms in our background job workers. Full table scan performance further degrades as the table size grows, so this query would have eventually consumed enough time inside the database to impact other queries that are servicing web requests. By optimizing this query, we’ve ensured that this process will never impact the user experience. Try it out now! You can see the real-time query statistics for your PlanetScale databases right now. Or you can sign up for PlanetScale and migrate your data to get more insight into your databases. Check it out and let us know what you think.]]> NoneSQL All the DevEx https://planetscale.com/blog/nonesql-all-the-devex 2021-08-27T15:36:52.435Z 2021-08-27T15:36:52.435Z Justin Gage Automatically copy migration data in PlanetScale branches https://planetscale.com/blog/automatically-copy-migration-data-in-planetscale-branches 2021-08-23T15:23:00.000Z 2021-08-23T15:23:00.000Z Taylor Barnett Building PlanetScale with PlanetScale https://planetscale.com/blog/building-planetscale-with-planetscale 2021-08-18T15:05:00.000Z 2021-08-18T15:05:00.000Z Iheanyi Ekechukwu database: <%= ENV['ENABLE_PSDB'] ? 'ourdatabase' : 'psdb_development' %> primary_replica: <<: *default port: <%= ENV['ENABLE_PSDB'] ? 3305 : nil %> database: <%= ENV['ENABLE_PSDB'] ? 'ourdatabase' : 'psdb_development' %> replica: true # Connect to the main production database and start the PlanetScale Proxy if Rails.env.production? PlanetScale.start( org: 'planetscale', db: 'ourdatabase', branch: 'main' ) elsif Rails.env.development? && ENV['ENABLE_PSDB'] PlanetScale.start(org: 'planetscale') end Applying the schema changes and creating a deploy request When it comes time to apply this migration, we run ENABLE_PSDB=1 bundle exec rake db:migrate, which then applies the schema change against the database branch. You can then either go to the PlanetScale application and open a deploy request from the branch itself, or use the CLI to create a deploy request like so:pscale deploy-request create ourdatabase add-audit-logs-table Usually, I visit the Deploy requests page within our database in the web application and copy the URL to the deploy request, or construct it myself from the deploy request number. The creator then pastes that URL into the body of their pull request on GitHub so reviewers can look at both side-by-side. The deploy request also shows the DDL statements (CREATE/ALTER/DROP) for each table changed in the migration, with a line-by-line diff, so everybody with access can clearly see what will happen. We have a dedicated Slack channel for pull requests, which usually helps decrease the turnaround time on reviews. Posting a link to the deploy request in Slack also doesn’t hurt, which helps decrease the review time since schema migrations tend to be fairly brief. Deploying the schema changes After a deploy request has been approved by a team member, the creator can deploy the schema changes to the main production branch by adding it to the deploy queue. The deploy queue enables multiple deployments to be queued up at once, so another teammate can also queue up their schema changes for deployment after the currently running one. If anything goes wrong during the migration (such as adding a NOT NULL constraint to a row that is NULL), the deployment will stop and show the relevant error. After fixing the error, the deployment can then be restarted. After the deployment is successfully completed, the GitHub pull request is merged and our application gets deployed to production with the new changes. If something needs to be added or changed in the schema, it’s just as easy to create another deploy request with the updates and push those changes to a new pull request. The beauty of this process is that it decouples the deployment of database schema changes from the application deployment process without needing a database administrator to handle it. Database migrations aren’t scary anymore The beauty of this process is that it decouples the deployment of database schema changes from the application deployment process without needing a database administrator to handle it. Additionally, these migration deployments come with no downtime or locking of production database tables. Since we’ve started using database branches and deploy requests to manage our schema in production, I feel empowered whenever I’m building new features or making schema changes. It’s no longer scary to make changes to the database, even after we’ve had some long-running (multi-day) migrations. We can see the operations that will occur directly in a deploy request before we deploy them. Non-blocking schema changes help us move fast and ship new features without breaking things or requiring a database administrator. …I feel empowered whenever I’m building new features or making schema changes. It’s no longer scary to make changes to the database… Whether you are a developer who likes to hack on side projects or part of an engineering team, I’d love for you to experience the joy of using PlanetScale. Sign up today and give us a shot. Happy hacking! P.S. We’re hiring! If you’re interested in being a part of the team that builds the best database for developers, take a look at our careers page!]]> Connect any MySQL client to PlanetScale using Connection Strings https://planetscale.com/blog/connect-any-mysql-client-to-planetscale-using-connection-strings 2021-08-16T20:03:00.000Z 2021-08-16T20:03:00.000Z Taylor Barnett PlanetScale on Vitess https://planetscale.com/blog/planetscale-on-vitess 2021-07-20T19:30:00.000Z 2021-07-20T19:30:00.000Z Deepthi Sigireddi Sam Lambert appointed new CEO of PlanetScale https://planetscale.com/blog/new-ceo-of-planetscale 2021-07-19T07:00:00.000Z 2021-07-19T07:00:00.000Z Jiten Vaidya The promises and realities of the relational database model https://planetscale.com/blog/the-realities-of-the-relational-database-model 2021-07-13T04:00:00.000Z 2021-07-13T04:00:00.000Z Shlomi Noach Integrating PlanetScale with Vercel in a few steps https://planetscale.com/blog/planetscale-vercel-integration 2021-07-01T16:00:00.000Z 2021-07-01T16:00:00.000Z Nick Van Wiggeren Serverless finally has a database https://planetscale.com/blog/serverless-finally-has-a-database 2021-05-24T17:00:00.000Z 2021-05-24T17:00:00.000Z Sam Lambert Non-Blocking Schema Changes https://planetscale.com/blog/non-blocking-schema-changes 2021-05-20T18:45:00.000Z 2021-05-20T18:45:00.000Z Lucy Burns Announcing PlanetScale: The database for developers. https://planetscale.com/blog/announcing-planetscale-the-database-for-developers 2021-05-18T17:10:00.000Z 2021-05-18T17:10:00.000Z Sam Lambert Announcing Vitess 9.0 https://planetscale.com/blog/announcing-vitess-9 2021-01-27T19:30:00.000Z 2021-01-27T19:30:00.000Z Alkin Tezuysal Announcing Vitess 8.0 https://planetscale.com/blog/announcing-vitess-8 2020-10-27T17:30:00.000Z 2020-10-27T17:30:00.000Z Alkin Tezuysal Pitfalls of isolation levels in distributed databases https://planetscale.com/blog/pitfalls-of-isolation-levels-in-distributed-databases 2020-10-04T18:30:00.000Z 2020-10-04T18:30:00.000Z Sugu Sougoumarane MySQL semi-sync replication: durability consistency and split brains https://planetscale.com/blog/mysql-semi-sync-replication-durability-consistency-and-split-brains 2020-10-02T04:00:00.000Z 2020-10-02T04:00:00.000Z Shlomi Noach Consensus algorithms at scale: Part 3 - Use cases https://planetscale.com/blog/consensus-algorithms-at-scale-part-3 2020-09-26T04:00:00.000Z 2020-09-26T04:00:00.000Z Sugu Sougoumarane Orchestrator failure detection and recovery: New Beginnings https://planetscale.com/blog/orchestrator-failure-detection 2020-09-19T04:00:00.000Z 2020-09-19T04:00:00.000Z Shlomi Noach Consensus algorithms at scale: Part 2 - Rules of consensus https://planetscale.com/blog/consensus-algorithms-at-scale-part-2 2020-09-09T04:00:00.000Z 2020-09-09T04:00:00.000Z Sugu Sougoumarane On joining PlanetScale and the vision of open source database infrastructure https://planetscale.com/blog/on-joining-planetscale-and-the-vision-of-open-source-database-infrastructure 2020-09-01T19:30:00.000Z 2020-09-01T19:30:00.000Z Shlomi Noach Consensus algorithms at scale: Part 1 - Introduction https://planetscale.com/blog/consensus-algorithms-at-scale-part-1 2020-08-28T07:00:00.000Z 2020-08-28T07:00:00.000Z Sugu Sougoumarane Learn Horizontal Scaling on PlanetScaleDB with Vitess — Rate Puppies in a Rust app with Sharded MySQL Database https://planetscale.com/blog/learn-horizontal-scaling-on-planetscaledb-with-vitess-rate-puppies-in-a-rust-app-with-sharded-mysql-database 2020-08-14T19:45:00.000Z 2020-08-14T19:45:00.000Z Jiten Vaidya Announcing Vitess 7 https://planetscale.com/blog/announcing-vitess-7 2020-07-28T07:00:00.000Z 2020-07-28T07:00:00.000Z Deepthi Sigireddi Debunking 3 myths about Vitess fault tolerance https://planetscale.com/blog/debunking-3-myths-about-vitess-fault-tolerance 2020-06-10T21:00:00.000Z 2020-06-10T21:00:00.000Z Abhi Vaidyanatha Announcing Vitess 6 https://planetscale.com/blog/announcing-vitess-6 2020-04-29T07:01:00.000Z 2020-04-29T07:01:00.000Z Morgan Tocker ACID Transactions are not just for banks — the Vitess approach https://planetscale.com/blog/acid-transactions-are-not-just-for-banks-vitess-approach 2020-04-29T07:00:00.000Z 2020-04-29T07:00:00.000Z Jiten Vaidya Videos: Intro to Vitess—its powerful capabilities and how to get started https://planetscale.com/blog/videos-intro-to-vitess-its-powerful-capabilities-and-how-to-get-started 2020-04-23T07:00:00.000Z 2020-04-23T07:00:00.000Z Abhi Vaidyanatha PlanetScale migrates open source Vitess test suite from Python to Go https://planetscale.com/blog/planetscale-migrates-open-source-vitess-test-suite-from-python-to-go 2020-03-20T07:00:00.000Z 2020-03-20T07:00:00.000Z Deepthi Sigireddi