MemSQL Developer Blog

MemSQL ships 2.0. Scales in-memory database across hundreds of nodes, thousands of cores.

MemSQL Engineering — Tue, 23 Apr 2013 09:17:24 +0000

When we started MemSQL a little over two years ago, our goal was to deliver the fastest OLTP database ever. Inspired by the scale and architectures we saw at Facebook, we hoped to help every enterprise leverage in-memory technologies similar to those that leading web companies use.

As we worked with our early customers, we saw that an in-memory solution could provide the greatest value by enabling users to analyze real-time and recent historical data together. Customers like Zynga and Morgan Stanley not only wanted to quickly commit transactions to the database, they also wanted instant answers to questions about how their real-time data compared to historical data. This inspired us to build something new – a solution that supports highly concurrent transactional and analytical workloads at Big Data scale.

That brings us to today. We’re proud to announce that MemSQL’s real-time analytics platform is available for download. This is the first generally available version of MemSQL that scales horizontally on commodity hardware. It provides the blazing fast performance for which MemSQL is known, and now does it at Big Data scale. Customers have deployed MemSQL across hundreds of nodes and dozens of terabytes of data, and we’ve tested at even greater volumes and velocities. (Check out our calculator to get an idea of the number of reads and writes you can perform depending on the size of your cluster.)

This is also the first version to include MemSQL Watch, a visual web-based interface for monitoring and managing your cluster. We expect this to be the beginning of our foray into real-time visualizations as many of our customers look to operationalize their analytics.

Deploying a database can be difficult, so we’ve made it as simple as possible. Download MemSQL for free on our site and take it for a spin. You’ll definitely be impressed by the performance, but you’ll also be impressed by what’s missing:

Batched loading – Don’t wait until the middle of the night to refresh your reports.
Complicated programming languages (and a limited talent pool) – Use SQL for real-time analytics.
An expensive, proprietary box (and a plan to rip and replace it in a few years) – Scale incrementally on commodity hardware.
A lengthy implementation cycle – Launch your first MemSQL instance in minutes in the cloud.

We’re proud of the progress our engineering team has made building out an enterprise-grade software solution. Stay tuned to this blog to learn more about the real-time analytics challenges we are helping customers conquer. More to come.

Practical Techniques to Achieve Quality in Large Software Projects

MemSQL Engineering — Wed, 10 Apr 2013 17:27:11 +0000

High quality is hard to achieve and very expensive, but it’s worth every penny and must be taken extremely seriously. There is no silver bullet – just lines of defense. The good news is with the proper lines of defense, quality becomes incremental. It only goes up with every release. With enough test coverage and quality tools you can substantially increase the quality of your product and protect it from regressions.

When you embark upon a large software project you need figure out how to think about quality. And you should start with testing. Not all tests are created equal. Here are the most common types:

Unit tests
Invariant tests (asserts, etc)
Functional tests
Stress tests
Performance tests
Scalability tests
Customer workload based testing
Scenario based testing

In this blog post we will discuss test coverage as well as how to manage the long tail of rarely surfaced bugs using force multipliers - a slew of engineering techniques that transform our existing test suite to produce new ways of testing the product. Each of these has been a small “aha” moment and yielded good results.

Put Your Test Plan on Paper: The Test Matrix

With testing, one of the hardest things is deciding exactly what to test. Every test you create or run has a cost, and this is where a tester’s intuition shines. When your product has a lot of features it’s easy to poke around, but the real question is how can you do it in a systematic way.

It’s important to track which areas of the product are tested well and which are not, and this is made all the more difficult by the many moving parts of the codebase. Here are a few MemSQL examples:

Correctness of read queries in the presence of write queries
Correctness of replication in the presence of alter table

It helps tremendously to test combinations of features. It’s especially hard because different people or even teams could work on those features and the problem may only arise when both features are involved at the same time.

The number of possible combinations explodes very quickly. However, we’ve noticed that in practice the majority of bugs surface in the combination of just two features.

So, one way to tackle the combinatorial explosion is to only consider pairs of features and write tests for every pair. You can also employ n-wise testing and use a different combination every time.

On top of that you should also think about dimensions. A classical test dimension for a SQL database is “all data types” or “all possible positions within a SQL query such as the where clause, select clause, group by, etc.” When testing a specific feature you want to test it with every dimension.

We wrote down all feature pairs and dimensions into a giant spreadsheet. After that we fanned out the test work to the engineering team to color the matrix green.

Invest Into A Stellar Test Execution System

You can write a lot of tests, but it is continuous integration testing that allows developers to make far-reaching changes without a paralyzing fear of undetected breaks^[1]. Such integration testing should be fast and scalable. You need stellar test infrastructure to frictionlessly add, manage, and run these tests. With a well-built execution system, the number of functional tests can grow safely and quickly.

Building good test infrastructure is similar to writing a consumer product for your engineering team. The easier it is to produce tests, the more the engineering team is engaged and more tests are running against your system every day. In the ideal world, when you modify a single line of code you should instantly know what tests you just broke. Practically it’s not always possible because of the computational complexity of test runs, but you should push towards getting instant feedback as much as possible.

A perfect test execution system would run the entire test suite after every push. We used to run every test before commits on each developer’s machine. When that started taking too long, we ran only a subset of tests locally and performed full test runs every night on a dedicated box. When the dedicated box couldn’t handle the load, we built a cloud-based test execution platform called “Psyduck.”

The Psyduck platform can run any subset of our tests against any version of our codebase. Furthermore, if you want to run tests against local code changes, Psyduck accepts patches, and so we enforce running Psyduck before pushing code. For example:

memcompute1:~/memsql/memsqltest nikita(arcpatch-D2497)$ ./psy test --filter=.nikita
Compressing patch...done.
Sending 5340 bytes to S3.....done.
patch id = 901be82c33ee4b07b142d102660e3206Psyduck uses a combination of in-house hardware along with hundreds of Amazon spot instances to run tests as fast as possible in parallel.

Last summer a MemSQL intern built a real-time visualization layer called Liveduck and it became an instant hit. This interface displays all ongoing and recently completed tests along with metadata such as the associated engineer and the pass/fail count.

Psyduck provides us with a quick metric on where we stand now and where we need to be at release. As we approach a release, the team pushes towards what we call “green Psyduck”. This means that all tests are passing.

Another property of an excellent test system is that every test must run the same on a developer’s machine as it does on the platform. Furthermore, each test should be self-contained so that it requires zero setup. This will greatly assist in test suite maintenance and accelerate failure investigation.

Force Multipliers

Every one of these tricks took MemSQL quality to the next level. Here is the list.

Transforms

Transforms allow to put a twist on all existing tests. By implementing a few functions in Python you can munge both the incoming query and the expected output. Then, you can run your entire functional test suite via a transform. MemSQL has many transforms. Here are few:

Replication transform. All the read queries are directed to a replication slave.
Subquery transform. Every read query is wrapped into subquery
Backup restore transform. As queries run, the transform performs backups and restore and makes sure the results of the queries stay the same

We have had internal transform competitions, where the goal is to write the transform which finds the most bugs. When you write transforms you feel like you are standing of the shoulders of giants. With a few lines of code, you can stress the system with thousands of new tests.

Random Query Generation

Random query generators produce SQL statements using a given grammar. The cool part is that it can read the bison grammar of the actual product and use it to generate random queries. There are a bunch of force multipliers applicable for random query generation:

Produce many different queries that give the same result; for example, identical queries over tables with different indexes and cross verify the results.
Build intelligence into the generator to gradually increase the complexity of the generated tests. This ensures that the system is robust against simple cases before you proceed into more complex cases. If you start with complex cases you won’t be able to converge to a quality system fast enough.
Once a bug is identified, the generator automatically reduces the complexity of the query to come up with the smallest possible reproducible test case. Random query generators can produce extremely intimidating queries. If the repro is left “as is” it would take a lot more time for a developer to investigate the root cause of the failure.

Functional Stress

Now that you have created a lot of functional tests, you can leverage them to run functional stress. The idea is that you take all these tests and randomly throw them at the system in parallel. When you do that you can’t really verify the test results, but all you care about is crashes and internal consistency checks.

Stack Hasher

This is a very neat trick. Every time there is a condition that is true 99.9% of the time (like memory allocations), you inject a failure once per unique call stack. So if you run a query “select * from t” the first time the server will throw an error with the first memory allocation. Then when you run this command again, the first memory allocation succeeds because this call stack has already been failed once (and saved in the stackhasher), but the next allocation fails, and so forth. If you can keep re-running a query until it succeeds, you can deterministically test each possible resource failure. We have a transform which does exactly this with every test in our code base, and it has proven invaluable.

Production failures

Even though you test relentlessly, you still need pathways for your customers to surface issues back to your engineering team for investigation. You want to introduce minidump and full core dump functionality, in addition to logging to correctly diagnose bugs.

Promote Transparency and Quality Culture Via Code Review

One of the first steps towards quality that we did was to have every change go through a code review. To assist with this process a good tool is a must. We use Phabricator because it integrates into our workflow and is fun and easy to use.

Code review helps promote a quality-oriented culture in the office. Engineers will trend towards producing better code and more tests to increase the number of positive comments they receive. Furthermore, having such a system in place helps new engineers learn the ropes faster.

Conclusion

It’s entirely worthwhile to invest into testing infrastructure from Day One. By doing so, you can achieve the following goals:

Using stellar tools and force multipliers to do more with fewer engineers and really push the long tail of bugs
Always be aware of the current state of the quality of the product
Quickly stabilize and ship the product!

[1] Steven Hemingray. Google

Common Pitfalls in Writing Lock-Free Algorithms

Pieguy — Wed, 27 Mar 2013 22:16:34 +0000

Formally, a multi-threaded algorithm is considered to be lock-free if there is an upper bound on the total number of steps it must perform between successive completions of operations.
The statement is simple, but its implications are deep – at every stage, a lock-free algorithm guarantees forward progress in some finite number of operations.
Deadlock is impossible.

The promise of a lock-free algorithm seems remarkable in theory.
Concurrent threads can modify the same object, and even if some thread or set of threads stalls or stops completely in the middle of an operation,
the remaining threads will carry on as if nothing were the matter.
Any interleaving of operations still guarantees of forward progress.
It seems like the holy grail of multi-threaded programming.

This stands in contrast to the traditional method of placing locks around "critical sections" of code.
Locking prevents multiple threads from entering these critical sections of code at the same time.
For highly concurrent applications, locking can constitute a serious bottleneck.
Lock-free programming aims to solve concurrency problems without locks. Instead, lock-free algorithms rely on atomic primitives such as the classic "compare-and-swap" which atomically performs the following:

1 bool CompareAndSwap(Value* addr, Value oldVal, Value newVal){
2     if(*addr == oldVal){
3         *addr = newVal;
4         return true;
5     }else{
6         return false;
7     }
8 }

The biggest drawbacks to using lock-free approaches are

Lock-free algorithms are not always practical.
Writing lock-free code is difficult.
Writing correct lock-free code is extremely difficult.

To illustrate the third point above, let’s break down a typical first attempt at writing a lock-free stack. The idea is to use a linked-list to store nodes, and use CompareAndSwap to modify the head of the list, in order to prevent multiple threads from interfering with each other.

To push an element, we first create a new node to hold our data.
We point this node at the current head of the stack, then use CompareAndSwap to point the head of the stack to the new element.
The CompareAndSwap operation ensures that we only change the head of the stack if our new node correctly points to the old head (since an interleaving thread could have changed it).

To pop an element, we snapshot the current head, then replace the head with the head’s next node.
We again use CompareAndSwap to make sure we only change the head if its value is equal to our snapshot.

The code, in C++

 1 template <class Entry>
 2 class LockFreeStack{
 3     struct Node{
 4         Entry* entry;
 5         Node* next;
 6     };
 7 
 8     Node* m_head;
 9 
10     void Push(Entry* e){
11         Node* n = new Node;
12         n->entry = e;
13         do{
14             n->next = m_head;
15         }while(!CompareAndSwap(&m_head, n->next, n));
16     }
17 
18     Entry* Pop(){
19         Node* old_head;
20         Entry* result;
21         do{
22             old_head = m_head;
23             if(old_head == NULL){
24                 return NULL;
25             }
26         }while(!CompareAndSwap(&m_head, old_head, old_head->next));
27 
28         result = old_head->entry;
29         delete old_head;
30         return result;
31     }
32 }

Unfortunately, this stack is riddled with errors.

Segfault

The Push() method allocates memory to store linked-list information, and the Pop() method deallocates the memory.
However, between the time a thread obtains a pointer to a node on line 22, and subsequently accesses the node on line 26, another thread could have removed and deleted the node, and the thread will crash.
A safe way to reclaim the memory is needed.

Corruption

The CompareAndSwap method makes no guarantees about whether the value has changed, only that the new value is the same as the old value.
If the value is snapshotted on line 22, then changes, then changes back to the original value, the CompareAndSwap will succeed. This is known as the ABA problem.
Suppose the top two nodes on the stack are A and C.
Consider the following sequence of operations:

Thread 1 calls pop, and on line 22 reads m_head (A), on line 26 reads old_head->next (C), and then blocks just before calling CompareAndSwap.
Thread 2 calls pop, and removes node A.
Thread 2 calls push, and pushes a new node B.
Thread 2 calls push again, and this time pushes a new node occupying the reclaimed memory from A.
Thread 1 wakes up, and calls CompareAndSwap.

The CompareAndSwap on line 26 succeeds even though m_head has changed 3 times, because all it checks is that old_head is equal to m_head.
This is bad because now m_head will point to C, even though B should be the new head.

Not lock-free

The C++ standard makes no guarantee that new and delete will be lock-free.
What good is a lock-free data structure that makes calls to non-lock-free library calls?
For our stack to be truly lock-free, we need a lock-free allocator.

Data races

When one thread writes a value to a location in memory, and another thread simultaneously reads from the same location,
the result is undefined in C++ unless std::atomic is used.
The reads and writes both need to be atomic, and C++ makes no such guarantee for primitive types.
Prior to C++11, it was a common practice to use volatile for atomic variables, however this approach is fundamentally flawed.

In this case, threads reading the head pointer constitute data races, both in Push and Pop, because another thread could be simultaneously modifying it.

Memory reordering

It is convenient to think of computers as being sequentially consistent machines,
where every load or store can be placed into a single total order that is consistent with every thread. At the very least, we would expect the “happens before” relationship to be transitive.
Unfortunately this is not the case, and in both theory and practice the following sequence of events is possible, with x and y initialized to 0:

thread 1	thread 2
print x x.store(1) print y	print y y.store(1) print x

thread 1

thread 2

print x

x.store(1)

print y

y.store(1)

print x

output:

thread 1	thread 2
x -> 0 y -> 0	y -> 0 x -> 0

thread 1

thread 2

x -> 0

y -> 0

x -> 0

This would seem to imply that the printing of x happened before the store of x, which happened before the printing of y, which happened before the store of y, which in turn happened before the printing of x. This is just one example of how different threads may see changes to memory occur in different orders.
Furthermore, it is possible for loads and stores to be reordered within a single thread, both by the compiler and by the hardware.
In the above lock-free stack a reordered store could result in another thread seeing a stale version of a node’s next pointer and subsequently dereferencing an invalid memory location.

How to write a correct lock-free stack

Most of the above problems have many different solutions.
Here I’ll describe the approaches I use at work for our lock-free stack.

Segfault

Before dereferencing a node, we need to make sure it cannot be deleted.
Each thread has a special globally visible pointer called a "hazard pointer".
Before accessing the head node, a thread will set its hazard pointer to point to that node.
Then, after setting the hazard pointer it makes sure the node is still the head of the stack.
If it is still the head, then it is now safe to access.
Meanwhile, other threads that may have removed the node from the stack cannot free the associated memory until no hazard pointer points to this node.
For efficiency, the deletion process is deferred to a garbage collection thread that amortizes the cost of checking hazard pointers.

Note that the ABA problem does not have any consequences here. It does not matter if the head changed between the two reads. As long as the hazard pointer has been continuously pointing to a node for some interval of time that includes a moment in which the node belonged to the stack, then it is safe to access the node.

Corruption

One way to avoid the ABA problem is to ensure that the head of the stack never has the same value twice.
We use "tagged pointers" to ensure uniqueness of head values.
A tagged pointer contains a pointer and a 64-bit counter.
Every time the head changes, the counter is incremented.
This requires a double-width CompareAndSwap operation, which is available on most processors.

Not lock-free

Instead of allocating a separate node to store each entry, the "next" pointer is embedded in the entry itself,
thus no memory allocation is needed in order to push it onto the stack.
Furthermore, we may reuse the same node multiple times, recycling rather than deleting.
Requiring an extra 8 bytes per entry is an unattractive property if you want a stack for small data structures,
but we only use the stack for large items where the 8 bytes is insignificant.

Data races

We currently use boost::atomic for this.
Our compiler is gcc 4.6, which currently supports std::atomic, but its implementation is much less efficient than boost’s. In gcc 4.6, all operations on atomic variables apply expensive memory barriers, even when a memory barrier is not requested.

Memory reordering

C++11 provides a new memory model and memory ordering semantics for atomic operations that control reordering.
For our lock-free stack to work correctly, CompareAndSwap needs to be performed with sequentially consistent memory semantics (the strongest kind).
Sequential consistency means that all such operations can be put into a total order in which all threads observe the operations.
There is a very easy to miss race condition (that went unnoticed in our code for months) involving the aforementioned hazard pointers.
It turns out that hazard pointers also need to be assigned using sequentially consistent memory semantics.

Here’s the case that broke our stack:

Thread 1, in preparation for a Pop operation, reads the head of the stack.
Thread 1 writes the current head to its hazard pointer (using only release semantics, which are weaker than sequentially consistent semantics).
Thread 1 reads the head of the stack again.
Thread 2 removes head from the stack, and passes it to the garbage collector thread (using sequentially consistent memory semantics).
The garbage collector scans the hazard pointers,
and (because the assignment was not done with sequentially consistent memory semantics) is not guaranteed to see thread 1′s hazard pointer pointing to the node.
The garbage collector deletes the node
Thread 1 dereferences the node, and segfaults.

With sequentially consistent memory semantics applied to both the hazard pointer assignment and head modification,
the race condition cannot happen.
This is because for any two sequentially consistent operations, all threads must see them happen in the same order.
So if thread 2 removing the pointer happens first, thread 1 will see a different value on its second read and not attempt to dereference it.
If thread 1 writes to its hazard pointer first, the garbage collector is guaranteed to see that value and not delete the node.

Performance

Now that we’ve solved the issues with our stack, let’s take a look at a benchmark.
This test was performed on an 8 core Intel(R) Xeon(R) machine.
The workload consisted of random operations, with Push and Pop equally likely.
The threads were not throttled in any way – all threads performed as many operations as the machine could handle.

Well, that’s not very impressive.

The stack has a single point of contention, and is limited by the rate at which we can modify the head of the stack.
As contention increases, so does the number of failed CompareAndSwap operations.
A simple yet highly effective way to reduce the number of failed operations is simply to sleep after each failed operation.
This naturally throttles the stack to a rate it can effectively handle.
Below is the result of adding "usleep(250)" after any failed operation.

That’s right, the addition of a sleep improved throughput by roughly 7x! Furthermore, the addition of usleep(250) reduced processor utilization. Take a look at the output from HTOP while running each benchmark for 16 threads.

Locked stack:

Lock-free stack, no sleep:

Lock-free stack with usleep(250):

So lock-free is better, right? Well, the lock can be improved as well. Instead of using std::mutex, we can use a spinlock that performs the same usleep(250) each time it fails to acquire the lock. Here are the results:

Results

Under high contention, additional threads dramatically reduced overall throughput. The addition of a sleep reduced contention, and consequently both increased throughput while decreasing processor utilization.
Beyond 3 threads all stacks showed a relatively level performance.
None of the multi-threaded stacks performed better than a single-threaded stack.

Conclusion

Lock-free guarantees progress, but it does not guarantee efficiency. If you want to use lock-free in your applications, make sure that it’s going to be worth it – both in actual performance and in added complexity. The lock-free stack presented at the beginning of this article contained a multitude of errors, and even once the errors were fixed it performed no better than the much simpler locked version. Avoid trying to write your own lock-free algorithms, and look for existing implementations instead.

Code

Lock-based.

 1 #include 
 2 #include 
 3 
 4 template<class T>
 5 class LockedStack{
 6 public:
 7     void Push(T* entry){
 8         std::lock_guard<std::mutex> lock(m_mutex);
 9         m_stack.push(entry);
10     }
11 
12     // For compatability with the LockFreeStack interface,
13     // add an unused int parameter.
14     //
15     T* Pop(int){
16         std::lock_guard<std::mutex> lock(m_mutex);
17         if(m_stack.empty()){
18             return nullptr;
19         }
20         T* ret = m_stack.top();
21         m_stack.pop();
22         return ret;
23     }
24 
25 private:
26     std::stack<T*> m_stack;
27     std::mutex m_mutex;
28 };

Lock-free. The garbage collection interface is omitted, but in real applications you would need to scan the hazard pointers before deleting a node. The cost of the scan can be amortized by waiting until you have many nodes to delete, and snapshotting the hazard pointers in some data structure with sublinear search times.

 1 class LockFreeStack{
 2 public:
 3     // The elements we wish to store should inherit Node
 4     //
 5     struct Node{
 6         boost::atomic<Node*> next;
 7     };
 8 
 9     // Unfortunately, there is no platform independent way to
10     // define this class.  The following definition works in
11     // gcc on x86_64 architectures
12     //
13     class TaggedPointer{
14     public:
15         TaggedPointer(): m_node(nullptr), m_counter(0) {}
16 
17         Node* GetNode(){
18             return m_node.load(boost::memory_order_acquire);
19         }
20 
21         uint64_t GetCounter(){
22             return m_counter.load(boost::memory_order_acquire);
23         }
24 
25         bool CompareAndSwap(Node* oldNode, uint64_t oldCounter, Node* newNode, uint64_t newCounter){
26             bool cas_result;
27             __asm__ __volatile__
28             (
29                 "lock;"           // This makes the following instruction atomic (it is non-blocking)
30                 "cmpxchg16b %0;"  // cmpxchg16b sets ZF on success
31                 "setz       %3;"  // if ZF set, set cas_result to 1
32 
33                 : "+m" (*this), "+a" (oldNode), "+d" (oldCounter), "=q" (cas_result)
34                 : "b" (newNode), "c" (newCounter)
35                 : "cc", "memory"
36             );
37             return cas_result;
38         }
39     private:
40         boost::atomic<Node*> m_node;
41         boost::atomic<uint64_t> m_counter;
42     }
43     // 16-byte alignment is required for double-width
44     // compare and swap
45     //
46     __attribute__((aligned(16)));
47 
48     bool TryPushStack(Node* entry){
49         Node* oldHead;
50         uint64_t oldCounter;
51 
52         oldHead = m_head.GetNode();
53         oldCounter = m_head.GetCounter();
54         entry->next.store(oldHead, boost::memory_order_relaxed);
55         return m_head.CompareAndSwap(oldHead, oldCounter, entry, oldCounter + 1);
56     }
57 
58     bool TryPopStack(Node*& oldHead, int threadId){
59         oldHead = m_head.GetNode();
60         uint64_t oldCounter = m_head.GetCounter();
61         if(oldHead == nullptr){
62             return true;
63         }
64         m_hazard[threadId*8].store(oldHead, boost::memory_order_seq_cst);
65         if(m_head.GetNode() != oldHead){
66             return false;
67         }
68         return m_head.CompareAndSwap(oldHead, oldCounter, oldHead->next.load(boost::memory_order_acquire), oldCounter + 1);
69     }
70 
71     void Push(Node* entry){
72         while(true){
73             if(TryPushStack(entry)){
74                 return;
75             }
76             usleep(250);
77         }
78     }
79 
80     Node* Pop(int threadId){
81         Node* res;
82         while(true){
83             if(TryPopStack(res, threadId)){
84                 return res;
85             }
86             usleep(250);
87         }
88     }
89 
90 private:
91     TaggedPointer m_head;
92     // Hazard pointers are separated into different cache lines to avoid contention
93     //
94     boost::atomic<Node*> m_hazard[MAX_THREADS*8];
95 };

The code is also available on github (if you want to test it locally): https://github.com/memsql/lockfree-bench

Where Should TopCoders Work. One Year Later.

MemSQL Engineering — Sun, 23 Sep 2012 21:39:26 +0000

A little over a year ago we published a blog post “Where Should Top Coders Work?”. In a few weeks we were approached by topcoder.com and offered to sponsor TopCoder Open. We hired two amazing folks after that event, and two more joined as interns. With a handful of red TopCoders working on the engine, we’ve had a great relationship with the TopCoder community.

When I published the original blog post, I had absolute faith into the TopCoder brand, but we had nothing to really back it up. Since a year has passed, I thought it’d be worthwhile to share what our Top Coders have been working on.

What have TopCoders built at MemSQL?

Here at MemSQL, engineers have a lot of freedom in deciding what to work on. Instead of saying “we have a great team solving hard problems,” I’d like to share what each TopCoder has worked on since joining the company.

Pieguy. David fell in love with concurrency and lock-free data structures. He has built/improved our entire lock-free story. David is also great at finding tough race conditions by studying the code and investigating various concurrency scenarios including finding a bug in a well-known, lock-free paper (http://www.research.ibm.com/people/m/michael/podc-1996.pdf). TopCoder trains people to read and understand code very well.

Nika. Nika tackled the challenges in implementing joins and subqueries, which included building an updated optimizer, an artificial intelligence component that decides what indexes and what join order to use in a complex query.

Momchil. As an intern, Momchil worked on various components of the codebase, but he really got hooked working on distributed query execution. By the end of his internship MemSQL was able to run queries like the one below in a distributed environment over a 100 node cluster (actual use case):

select x.i, x.a, x.b, o.c, o.x from (select * from t) as x left join (select c, count(*) as x from ob group by c) o on x.a = o.c

Check out Momchil’s blog in which he writes about his experience at MemSQL.

Decowboy. While interning at MemSQL, Jelle worked on replication. He helped with the robust design and implemented major components of the feature. This is a highly technical project that involved transaction log shipping and recovery on the slave, failover, and resiliency to all kinds of DDL (data definition language) such as create/drop table and create drop database, etc.

SkidanovAlex. Alex is the first MemSQL TopCoder and he has contributed to all the major components of MemSQL. He has been a crucial contributor to MemSQL’s distributed product.

The list can go on and it’s just the highlights of what the TopCoder crew has built.

What can TopCoders learn at MemSQL?

Databases are complex at every level: storage engines, optimizers, parsers, distributed systems – things most regular developers don’t work with on a day-to-day basis.

In a famous blog post (http://herbsutter.com/welcome-to-the-jungle/), Herb Sutter claims:

Applications will need to be at least massively parallel, and ideally able to use non-local cores and heterogeneous cores.
Efficiency and performance optimization will get more, not less, important.
Programming languages and systems will increasingly be forced to deal with heterogeneous distributed parallelism.

MemSQL is at the forefront of building massively parallel software. For engineers who possess a solid foundation in problem solving with algorithms and data structures, MemSQL is a great place to build these skills.

Why do TopCoders succeed at MemSQL?

Apart from the obvious such as an incredibly high IQ, excellent problem solving skills, and proficiency in algorithms, there is one trait that we find very attractive: TopCoders love to code. If you code a lot every day it generates a compound effect. You are getting much better and more productive.

Startup vs Big Company

Growth and opportunity are crucial for a successful engineering culture. Unfortunately, big company environments often hinder growth potential because of bureaucracy and a lack of emphasis on engineering culture. We are lucky to have very hard problems in spades and have no bureaucratic roadblocks for a TopCoder to grow and advance his or her career.

Are we going to hire more TopCoders?

We are happy to announce that we are sponsoring TopCoder Open once again and to keep our doors open for more amazing engineers coming out of this unique community.

Being a TopCoder is not a prerequisite for working at MemSQL, but being a great engineer is. Please find us at careers@memsql.com

MemSQL: My First Real Startup Experience

momchil — Tue, 11 Sep 2012 02:33:17 +0000

This is reposted from Momchil Tomov’s blog. Momchil was part of the first summer batch of MemSQL Interns.

After seemingly stumbling into their office by accident, getting interviewed on a lark, and receiving an offer as my Christmas gift, I kept an open mind for what to expect from MemSQL. The one-year-old YC alum was set to build the world’s fastest database, leaving competitors like MySQL and MongoDB in the dust. Very ambitious indeed.

From day one, I was thrown in the fire pit. Someone had to finish the Workload Simulator, an important developer tool, before the launch. Everyone was super busy fixing bugs and polishing the product, so I had to do a quick Javascript/Flask/Python crash course and jump on it. After eight long nights, a thousand lines of code, and a huge amount of support from Ankur, my mentor, the Workload Simulator was finished to coincide with launch.

I spent the next month working on code generation: writing code that compiles SQL to C++, the core strength of MemSQL. Once I got acquainted with the codebase, Ankur decided to step up my game with data compression. I had to find a way to compress 10 TB of random strings down to at most 5 TB without compromising the speed of the transactions, and I had two weeks to do so. After some research, experimentation, and lots of dumb luck, I managed to bring it down to 2 TB. It required all the MemSQL knowledge I had obtained thus far, including things I learned during my interview.

Just as I finished the first prototype, I joined the team working on sharding. Sharding is what enables MemSQL to get distributed across multiple machines, an important feature of every scalable database and also an important feature for our biggest clients. I was onboarded on Sunday, began work on Monday, and shipped my first code review on Tuesday. One of the dev leads, Ankur, gauged my progress, and kept assigning me more and more sharding tasks, slowly walking me through the implementation of Distributed MemSQL. Just as a father silently lets go of the back seat of his kid’s bicycle, he slowly backed away and moved on to testing and bug fixing. Before realizing it, I found myself working on the team with no safety wheels, full speed ahead, pushing the forefront of MemSQL. I was no longer an intern. I was a full-time MemSQL engineer.

They say Princetonians work hard and play really really hard. If that’s true, then MemSQL felt even more like home. From the extraordinarily delicious meals prepared by our in-house chef Daniel, to the post-launch Las Vegas Celebration Trip, to the relaxing Saturdays on Vasili’s boat underneath the Golden Gate, the MemSQL team knows how to have a good time. As MemSQL is growing, I sincerely hope they maintain their great culture of ridiculous engineering and chill free time. I had an amazing summer many thanks to the incredible people at 380 10th Street. I wish Eric, Nikita, Marko, Vasili, Alex, Masha, Nika, Pieguy, Adam, Ankur, Daniel, et al. all the best and hope they build a company 300x better than the rest!

Go MemSQL!

Excellent Post On MemSQL Architecture

MemSQL Engineering — Wed, 15 Aug 2012 02:28:15 +0000

http://highscalability.com/blog/2012/8/14/memsql-architecture-the-fast-mvcc-inmem-lockfree-codegen-and.html

Loading half a billion records in 40 minutes

MemSQL Engineering — Mon, 13 Aug 2012 18:58:19 +0000

Disclaimer. This will not be an apples to apples comparison with derwik, since we obviously don’t have the same dataset, and we need a much bigger machine to load everything into memory. But I believe this experiment will get the point across. So without further ado, let’s go through the steps.

Adam Derewecki wrote a cool post about his experience loading half a billion records into MySQL. MemSQL is a MySQL-compatible drop-in replacement that is built from the ground up to run really fast in memory.

Disclamer: this isn’t an apples-to-apples comparison with derwik since we don’t have his dataset and need a much beefier machine to load everything into memory.

Schema

Here is the schema for the table. Note that the table has three indexes. On top of it memsql will automatically generate a primary index under the covers. We won’t be disabling keys.

drop table if exists store_sales_fact;
create table store_sales_fact( 
       date_key                smallint not null,
       pos_transaction_number  integer not null,
       sales_quantity          smallint not null,
       sales_dollar_amount     smallint not null,
       cost_dollar_amount      smallint not null,
       gross_profit_dollar_amount smallint not null,
       transaction_type        varchar(16) not null,
       transaction_time        time not null,
       tender_type             char(6) not null,
       product_description     varchar(128) not null,
       sku_number              char(12) not null,
       store_name              char(10) not null,
       store_number            smallint not null,
       store_city              varchar(64) not null,
       store_state             char(2) not null,
       store_region            varchar(64) not null,
       key date_key(date_key),
       key store_region(store_region),
       key store_state(store_state)
);

Hardware

We have a very cool machine, with 64 cores and 512 GB of RAM, from Peak Hosting . You can rent one for yourself for a little under two grand a month. They were kind enough to give it to us to use for free. Here is a spec of one core.

vendor_id       : AuthenticAMD
cpu family      : 21
model           : 1
model name      : AMD Opteron(TM) Processor 6276
stepping        : 2
cpu MHz         : 2300.254
cache size      : 2048 KB

You read that correctly, this machine has sixty-four 2.3 GHz cores and 512 GB of RAM or almost 8 times the largest memory footprint available in the cloud today, all on dedicated hardware with no virtualization overhead or resource contention with other unknown third parties.

Loading efficiently

Loading data efficiently is actually not that trivial. The best way of doing it with MemSQL is to use as much CPU as you can get. Here are a few tricks that can be applied.

1. Multi-inserts

We can batch inserts into 100 row multi-inserts. This will reduce the number of network roundtrips. Each roundtrip now accounts for 100 rows instead of one. Here is what multi-inserts look like.

insert into store_sales_fact values('1','1719','4','280','97','183','purchase','14:09:10','Cash','Brand #6 chicken noodle soup','SKU-#6','Store71','71','Lancaster','CA','West'),
('1','1719','4','280','97','183','purchase','14:09:10','Cash','Brand #5 golf clubs','SKU-#5','Store71','71','Lancaster','CA','West'),
('1','1719','4','280','97','183','purchase','14:09:10','Cash','Brand #4 brandy','SKU-#4','Store71','71','Lancaster','CA','West'),

2. Load in parallel.

Our customer has a sample file of 510,593,334 records. We can use the command line mysql client to pipe this file into MemSQL, but this would not leverage all the cores available in the system. So ideally we should spit the file to at least as many chunks as there are CPUs in the system.

3. Increase granularity

Splitting the file into 64 big chunks will introduce a data skew. The total data load time will be the time the slowest thread loads the data. To address this problem we will split the file into thousands of chunks. And every time a thread frees up we will start loading another chunk. So we split the file into 2000 chunks.

 1 -rw-r--r--  1  35662844 2012-06-06 11:42 data10.sql
 2 -rw-r--r--  1  35651723 2012-06-06 11:42 data11.sql
 3 -rw-r--r--  1  35658433 2012-06-06 11:42 data12.sql
 4 -rw-r--r--  1  35663665 2012-06-06 11:42 data13.sql
 5 -rw-r--r--  1  35667480 2012-06-06 11:42 data14.sql
 6 -rw-r--r--  1  35659549 2012-06-06 11:42 data15.sql
 7 -rw-r--r--  1  35661617 2012-06-06 11:42 data16.sql
 8 -rw-r--r--  1  35650414 2012-06-06 11:42 data17.sql
 9 -rw-r--r--  1  35661625 2012-06-06 11:42 data18.sql
10 -rw-r--r--  1  35667634 2012-06-06 11:42 data19.sql
11 -rw-r--r--  1  35662989 2012-06-06 11:42 data1.sql

4. Load script. The load script uses python multiprocessing library to load data efficiently.

 1 import re
 2 import sys
 3 import os
 4 import multiprocessing
 5 import optparse
 6
 7 parser = optparse.OptionParser()
 8 parser.add_option("-D", "--database", help="database name")
 9 parser.add_option("-P", "--port", help="port to connect. use 3306 to connect to memsql and 3307 to connect to mysql", type="int")
 10 (options, args) = parser.parse_args()
 11
 12 if not options.database or not options.port:
 13     parser.print_help()
 14     exit(1)
 15
 16 total_files = 2000
 17
 18 def load_file(filename):
 19     try:
 20         print "loading from cpu: %d" % os.getpid()
 21         query = 'mysql -h 127.0.0.1 -D %s -u root -P %d < %s' % (options.database, options.port, filename)
 22         print query
 23         os.system(query)
 24         print "done loading from cpu: %d" % os.getpid()
 25     except e as Exception:
 26         print e
 27         pass
 28
 29 os.system('echo "delete from store_sales_fact" | mysql -h 127.0.0.1 -u root -P %d' % options.port)
 30 p = multiprocessing.Pool(processes = 2*multiprocessing.cpu_count())
 31 for j in range(0, total_files):
 32     p.apply_async(load_file, ['data/data%d.sql' % j])
 33
 34 p.close()
 35 p.join()

5. Running it

I started loading by issuing the following command.

time python load.py -D test -P 3306

After we do this let’s start htop to check the processor saturation. It looks pretty busy .

MemSQL uses lockfree data structures that eliminate a lot of contention.

It took a bit of time, but the data has been loaded.

 real    39m21.465s
 user    33m53.210s
 sys     5m24.470s

The result is almost 200K inserts a second for a table with 4 indexes. The memory footprint of the memsql process is 267 Gb.

Running the same test against mysql

I would not be fair to skip comparison with mysql, particularly that it’s so easy to do since memsql uses mysql wire protocol and support mysql syntax. I used the my.cnf settings from http://derwiki.tumblr.com/post/24490758395/loading-half-a-billion-rows-into-mysql
and fired up the same command

time python load.py -D test -P 3307

real    536m0.376s
user    27m17.130s
sys     5m36.780s

I did not disable indexes for this run to make it fair compared to MemSQL.

Conclusion

With the Peak Hosting offering of 512Gb of RAM and memory optimized software like MemSQL you can save full business days in data loading and get immediate, valuable insight into your data.

Reading Between the Benchmarks: How MemSQL Designs for Speed and Durability

Jelle — Mon, 25 Jun 2012 21:11:24 +0000

Foreword

There’s been a lot of discussion recently about how MemSQL compares to MySQL and other databases. Prior to the recent controversy, we were planning to publish the post below, where we jump in and describe a workload that we ran against MemSQL, MySQL, and MongoDB, all configured for asynchronous durability, on a few different hardware scenarios. Before that, though, we’d like to address some of the comments.

First, we’ve built MemSQL with a sensible set of trade-offs around index performance and durability in mind. The customers that we’ve worked with and optimized for have indeed seen on average a 30 times improvement for their workloads. Of course, just the fact that their data now lives in memory is a major part of the boost. But an important point is that MemSQL’s ease-of-use is what gets them there in the first place. MemSQL helps to relieve the cost of engineering caching or NoSQL infrastructure that syncs back to a database like MySQL.

Of course, we must consider MySQL’s in-memory performance. Jelle, one of our interns, explores this below by tuning InnoDB to behave like MemSQL. But the short answer is that MemSQL is a lot faster. And that’s because the database has been engineered from the ground up to run in memory. Everything from the index data structures (lock-free skip lists and hash tables) to every detail of our durability (log and snapshot format, transaction buffer, etc.) was designed and built for the in-memory, archive-to-disk use case.

Even though MemSQL runs in memory, it logs transactions to disk as fast as the disk will write. MemSQL uses an in-memory transaction buffer that is flushed to disk in a separate log flusher thread. By the time a transaction commits in memory, it has been written to the in-memory buffer. If the amount of data in the buffer exceeds transaction-buffer MB, then writes block until there is space available. Both MongoDB and MySQL are often run with similar configurations.

We expose the performance of this component through the show status query as Transaction_buffer_wait_time. This value measures the cumulative amount of time transaction threads have blocked trying to insert into the transaction buffer. The default size of the transaction buffer, 128 MB, is a heuristic: most disks can easily sequentially write 128 MB/s, so you shouldn’t lose more than a second of data. Visit the durability page in our documentation for more information.

We haven’t yet encountered a client who has asked us to optimize MemSQL for synchronous durability. Nevertheless, we know how to improve it in our current design and can discuss this in a future post.

The post below is meant to be a seed for you to run your own experiments with MemSQL. Since we can’t cover every workload, hardware configuration, and database in a benchmark, we’ve built something simple that runs MemSQL, MySQL, and MongoDB and we hope it will enable the community to continue running its own experiments. The more data points we have, the more we can improve the product.

- MemSQL Engineering

When I joined MemSQL at the beginning of this summer, I decided to write a small benchmark to see on my own how fast MemSQL can be. One of MemSQL’s main strengths is its highly optimized lock-free skiplist implementation. This means that MemSQL should scale almost linearly on machines with many cores. To explore this, I wrote a benchmark comparing the performance of MemSQL, MongoDB, and MySQL. My benchmark simulates a simplified online multiplayer game, with the database responsible for tracking players, games and events. The code for the benchmark is available on github.com/memsql/bench and can easily be extended to other databases.

The databases

The two databases I compared against MemSQL are MySQL and MongoDB. MemSQL is wire compatible with MySQL, so a comparison is natural and very easy. MemSQL and MongoDB have radically different interfaces as MongoDB is a NoSQL database. However, MongoDB is a popular choice for social applications so considering its performance is worthwhile as well.

MemSQL is an in-memory database that stores all the contents of the database in RAM but backs up to disk. MongoDB and MySQL store their data on disk, though can be configured to cache the contents of the disk in RAM and asynchronously write changes back to disk. This fundamental difference influences exactly how MemSQL, MongoDB and MySQL store their data-structures: MemSQL uses lock-free skip lists and hash tables to store its data, whereas MongoDB and MySQL use disk-optimized B-trees.

Description of the benchmark

The benchmark tests performance of the database when used as the backend for an online turn based multiplayer game. It was inspired by mobile phone versions of games such as multiplayer chess. For the core game logic, the database stores players and games. To track statistics, the database also stores a log of all past game actions and average game lengths.

The database stores four kinds of objects:

Players. Every player has an ID, and two statistics: the number of games started and the number of games won.
Games. Each game has an ID, two players and a turn.
Game length statistics. We store a histogram of finished game lengths, composed of a game length and an integer counting the number of games.
Events. Finally, every action performed is stored as an up to 32-character description, together with a timestamp and a player ID.

The benchmark simulates many players simultaneously playing games. Several Python worker processes each simulate hundreds of players. Each worker continually picks a player and then simulates one player action. All actions are logged in the events table. This logic is implemented in benchmark.py, and executes according to the following diagram: To support this simulation, the database needs to support several operations: find all games a player is currently playing, create a game, store statistics, etc. In the benchmark, these operations are described in base_database.py. Database-specific implementations can be found in sql_database.py and mongo_database.py. Since MySQL and MemSQL are protocol compatible, sql_database.py implements the interface for both MySQL and MemSQL. The MongoDB implementation can be found in mongo_database.py.

To support efficient queries, I added indices on all player ID fields. To allow efficient exploration of past events, I also added an index to the timestamp column in the events table. In MongoDB, the timestamp column is integrated in the object ID and is thus simpler. Furthermore, in MongoDB, embedded documents can be used as an alternative to seperate queries with indices. In this benchmark, I could have stored the list of active games in each player object. However, since each game is used by two players, I decided against this approach as it would move all writes to the player table. That would be an undesirable situation as MongoDB has a per collection write lock.

Settings used

By default, MySQL flushes after every transaction. To make MySQL behave like MongoDB and MemSQL, I set up MySQL to use a transaction buffer. The settings different from the default Ubuntu settings are the following:

default-storage-engine=INNODB
innodb_buffer_pool_size = 4GB
innodb_log_file_size = 128M
innodb_log_buffer_size = 4M
innodb_flush_log_at_trx_commit=2
innodb_thread_concurrency=0
innodb_flush_method=O_DIRECT
innodb_file_per_table

MemSQL and MongoDB are running with their default configurations, which means both are durable and use a transaction buffer. This means that all databases are configured to behave the same way. I used the latest stable version of each database (MongoDB 2.0.6, MySQL 5.5.25, MemSQL 1b).

Results

I ran the benchmark for 10 minutes on several machines. I first ran it on my old MacBook Pro (running Ubuntu natively), which has a dual core Intel(R) Core(TM) 2 Duo T9400. After that, I tried a server with 8 Intel(R) Xeon(R) cores, each running at 2.4GHz. Finally, I tried a 24 core machine with AMD(R) Opteron(TM) processors, each core clocking in at a 1.9GHz. For the two and eight core machines, the benchmark client was running on the same server as the database. For the 24 core machine the benchmark was running on a different server, directly connected over gigabit ethernet. The benchmark is configured to use 140 worker processes and 4000 players per worker process. This can be changed in config.py. I achieved the following number of actions (as described above) processed per second:

In all of these configurations, MemSQL outperforms both MongoDB and MySQL. Going from 2 to 8 cores, MemSQL improves its performanced by a factor 5. MySQL is almost four times as fast, and MongoDB is almost three times as fast as in the 2 core scenario. When going from 8 to 24 cores, MemSQL performs more than 3 times faster. However, MongoDB and MySQL perform slower than before. To verify this effect, I limited MongoDB and MySQL to run on 8 of the 24 cores, and they performed about as well as they did running on all 24 cores.

In this benchmark, MemSQL is significantly faster on all fronts. On two cores, MemSQL benefits from its efficient query parsing and pre-compiled code per query. When scaling from two to eight cores, all three databases are able to take advantage of the faster hardware. However, MemSQL scales better thanks to its lock-free data-structures which allow all 8 cores to interact efficiently with the database at once. The 24 core machine has individual cores slower than the individual cores of the 8 core machine. MongoDB and MySQL are not capable of taking advantage of these extra cores; instead, their performance degrades due to the slower cores. MemSQL on the other hand has a speed increase, as it is able to take advantage of all the available cores.

As a final note, the performance of all databases does not change significantly over time. I sampled the number of actions per second every second over the 10 minute run on the 24 core machine, and performance doesn’t change:

Conclusion

In the end, this simulation is just that: a simulation. But it demonstrates that MemSQL is very fast and scales well for write-heavy workloads like this game. We want this code to serve as a template for benchmarking MemSQL and other databases.

We made a strong effort to configure MySQL and MongoDB fairly. These tests can also be run on a variety of different hardware scenarios, network configurations, and modified workloads. There are even tune-able parameters within the provided benchmark (number of worker processes, number of players). We are as interested as you are to see your findings with MemSQL against other databases and in other benchmarks.

To explore MemSQL on your own, visit the MemSQL download page and get the developer edition. If you choose to set up a MemSQL AMI on Amazon EC2, you can get the database up and running on an 8 core machine in less than 5 minutes. The code for this benchmark is available on github. Happy Benchmarking!

Where Should Top Coders Work?

Nikita Shamgunov — Mon, 18 Jun 2012 03:49:45 +0000

My career as a software engineer really began when I won a medal at the ACM ICPC programming contest in 2001. To place in the tournament, I had spent 24 hours traveling from Russia to Vancouver and back, just to spend 5 hours on the actual competition.

The rules are simple: you have 5 hours to solve up to 12 problems. For each problem you need to implement a small program in Java or C++ and send it to the jury. They compile it and run it through an extremely intensive set of tests. Only if it passes every test will the jury count your submission.

Even though it only marked the beginning of my career, it had taken 4 years of intense preparation to place at the tournament – 4 years of training, learning to think fast, practicing on weekends, and other sorts of mental gymnastics. ACM was a good school for me. I
loved it, but I wasn’t alone.

Is it really hard to become a Top Coder?

Becoming an ACP ICPC medalist in 2011 is about 20 times harder than it was in 2001. People who can do it now are coding machines and algorithm junkies. They practice every day on TopCoder, Google Code Jam, and Code Force to stay sharp. They memorize whole books filled with algorithms and equations in case they need a tool to solve a problem quicker than their opponent. It should be no surprise that this programming subculture has caught the attention of many great companies.

So where do programming contest winners go and work? It used to be Microsoft (where I started my career), then Google. Now it’s Facebook. These companies know how valuable top coders are. Microsoft, Google and Facebook wouldn’t be where they are without exceptional engineers.

How do Top Coders fare after graduation?

They fare very well and tend to build their careers at big companies. Many are attracted to the high salary right out of school. However, according to topcoder.com, many ultimately find work at large companies to be tedious and less challenging than the mental jujitsu of programming contests.

Large teams have the luxury of compartmentalizing a problem to reduce complexity. This creates an unfortunate side effect: these smaller problems just aren’t that interesting.

In addition, many ICPC champions soon miss the dynamic of small teams, the kind their experienced while training and competing during university.

As it so happens, there is a time in a company’s history when it’s perfect for a top coder to join – when the company’s just getting started.

Why do some Top Coders found/join an early-stage startup?

For the challenge, of course.

Just take a look at these real-life examples:

Adam D’Angelo was a finalist in the international Topcoder Collegiate Challenge in 2005. Later he was VP and CTO of Facebook and then left to found Quora.
Nikolai Durov is Employee #1 at the biggest Russian social network vkontakte. It beats Facebook on the Russian market. He could’ve gone to Google or stayed in academia, but he made a small bet that had lots of upside.
One of my team members Leonid Volkov went to an early stage company and built the “TurboTax of Russia”. He enjoyed an incredibly successful exit, and he’s since gone into politics.
Prasanna Sankaranarayanan is the founder of LikeALittle and was the highest ranked Top Coder in India. It took him one year at Microsoft to realize that the perfect job for a Top Coder is at an early stage startup.

Small startups offer Top Coders lots of responsibility and with it the trust and autonomy to solve tough problems.

Top Coders at MemSQL

Today, MemSQL has four Top Coders. I won a bronze medal in ACM ICPC in 2001, and Alex Skidanov was a Top Coder #13 in 2008 and ACM ICPC #3 Champion. A Top Coder would feel right at home at MemSQL. Here, we work on hard algorithmic and systems-level problems, distributed systems, and cloud infrastructure. We also now have Top Coders who ranked #4 and #8 in algorithms. Sometimes it’s scary to leave a big corporation, but the truth is that apart from fun, market pay, and the potential of a huge upside, a good early stage startup gives the kind of experience that makes a Top Coder extremely relevant in today’s tech industry.

If you’d like to learn more, shoot us an email at topcoders@memsql.com.

Welcome to the MemSQL Developer Blog!

MemSQL Engineering — Mon, 18 Jun 2012 03:41:08 +0000

We’ve been hard at work building the world’s fastest database, and now that we’re shipping MemSQL, we’re looking forward to having a bit more time to blog about some of the fundamentals around the MemSQL technology.

In the coming weeks, we’ll be publishing posts that cover an array of topics, including benchmarking, stress testing, database theories, algorithms, and more.

In the mean time, we encourage you to download MemSQL and have some fun doing your own benchmarking. We’ve also published a workload simulator on Github to help get you started in the right direction.

Thanks for dropping by and we look forward to a lot of great posts and discussions.

MemSQL Developer Blog

MemSQL ships 2.0. Scales in-memory database across hundreds of nodes, thousands of cores.

Practical Techniques to Achieve Quality in Large Software Projects

Put Your Test Plan on Paper: The Test Matrix

Invest Into A Stellar Test Execution System

Force Multipliers

Transforms

Random Query Generation

Functional Stress

Stack Hasher

Production failures

Promote Transparency and Quality Culture Via Code Review

Conclusion

Common Pitfalls in Writing Lock-Free Algorithms

Segfault

Corruption

Not lock-free

Data races

Memory reordering

How to write a correct lock-free stack

Segfault

Corruption

Not lock-free

Data races

Memory reordering

Performance

Results

Conclusion

Code

Where Should TopCoders Work. One Year Later.

What have TopCoders built at MemSQL?

What can TopCoders learn at MemSQL?

Why do TopCoders succeed at MemSQL?

Startup vs Big Company

Are we going to hire more TopCoders?

MemSQL: My First *Real* Startup Experience

Excellent Post On MemSQL Architecture

Loading half a billion records in 40 minutes

Schema

Hardware

Loading efficiently

1. Multi-inserts

2. Load in parallel.

3. Increase granularity

4. Load script. The load script uses python multiprocessing library to load data efficiently.

5. Running it

Running the same test against mysql

Conclusion

Reading Between the Benchmarks: How MemSQL Designs for Speed and Durability

The databases

Description of the benchmark

Settings used

Results

Conclusion

Where Should Top Coders Work?

Is it really hard to become a Top Coder?

How do Top Coders fare after graduation?

Why do some Top Coders found/join an early-stage startup?

Top Coders at MemSQL

Welcome to the MemSQL Developer Blog!

MemSQL: My First Real Startup Experience