Time to make SQL Server demos a little more fun. Now you can download a torrent of the SQL Server database version of the Stack Overflow data dump.
- Full size circa 2018/06: 38GB torrent (magnet.) Expands to a ~304GB SQL Server 2008 database. Because it’s so large, we distribute it with BitTorrent – if you’re new to that, keep reading below for more detailed torrent instructions.
- Mini size circa 2010/12: 1GB direct download, expands to a ~10GB database called StackOverflow2010. It’s the same database schema as the full size StackOverflow database, but it’s only got the 2008-2010 data.
- SuperUser.com 2017/12 database – 1GB backup file, expands to a ~3GB SQL Server 2008 database. SuperUser has the same database schema as Stack Overflow, but it’s just a much smaller database since the site has less activity.
As with the original data dump, this is provided under cc-by-sa 3.0 license. That means you are free to share this database and adapt it for any purpose, even commercially, but you must attribute it to the original authors (not us):
How to Get the Database
- Install a BitTorrent client – I recommend qBittorrent, an ad-free open-source client.
- Download & open this .torrent file – it’s a small metadata file that tells your BitTorrent client where to connect and start downloading the files.
- Wait. The big file may take a few hours to download depending on your internet connection and how many other people are seeding the torrent.
- Extract the .7Zip files with 7Zip – it will create the database MDF, NDFs (additional data files), LDF, and a Readme.txt file. Don’t extract the files directly into your SQL Server’s database directories – instead, extract them somewhere else first, and then move or copy them into the SQL Server’s database directories. (This just avoids permissions hassles.)
- Attach the database – it’s in Microsoft SQL Server 2008 format (2005 for the older torrents), so you can attach it to any 2008 or newer instance. It doesn’t use any Enterprise Edition features like partitioning or compression, so you can attach it to Developer, Standard, or Enterprise Edition. (If your SSMS crashes or throws permissions errors, you likely tried extracting the archive directly into the database directory, and you’ve got permissions problems on the data/log files.)
Please leave the torrent up and running – seeding the torrent helps other folks get it faster.
Why I’m Using BitTorrent
BitTorrent is a peer-to-peer file distribution system. When you download a torrent, you also become a host for that torrent, sharing your own bandwidth to help distribute the file. It’s a free way to get a big file shared amongst friends.
The download is relatively large, so it would be expensive for me to host on a server. For example, if I hosted it in Amazon S3, I’d have to pay around $1-$2 USD every time somebody downloaded the file. I like you people, but not quite enough to go around handing you dollar bills. (As it is, I’m paying for a seedbox to get this thing started.)
Some corporate firewalls understandably block BitTorrent because it can use a lot of bandwidth, and it can also be used to share pirated movies/music/software/whatever. If you have difficulty running BitTorrent from work, you’ll need to download it from home instead.
What’s Inside the StackOverflow Database
I want you to get started quickly while still keeping the database size small, so:
- All tables have a clustered index on Id, an identity field
- No other indexes are included (nonclustered or full text)
- The log file is small, and you should grow it out if you plan to build indexes or modify data
- It only includes StackOverflow.com data, not data for other Stack sites
To get started, here’s a few helpful links:
- This Meta.SE post explains the database schema.
- If you want to learn how to tune queries, Data.StackExchange.com is a fun source for queries written by other people.
- For questions about the data, check the data-dump tag on Meta.StackExchange.com.
Past Versions
I also keep past versions online too in case you need to see a specific version for a demo.
- 2017-12 – 19GB torrent (magnet.) Expands to a ~137GB SQL Server 2008 database.
- 2017-08 – 16GB torrent (magnet), 122GB SQL Server 2008 database. Starting with this version & newer, each table’s Id fields are identity fields. This way we can run real-life-style insert workloads during my Mastering Query Tuning class. (Prior to this version, the Id fields were just INTs, so you needed to select the max value or some other trick to generate your own Ids.)
- 2017-06 – 16GB torrent (magnet), 118GB SQL Server 2008 database. Starting with this torrent & newer, I broke this up into multiple SQL Server data files, each in their own 7z file, to make compression / decompression / distribution a little easier. You need all of those files to attach the database.
- 2017-01 – 14GB torrent (magnet), 110GB SQL Server 2008 database
- 2016-03 – 12GB torrent (magnet), 95GB SQL Server 2005 database
- 2015-08 – 9GB torrent (magnet), 70GB SQL Server 2005 database
66 Comments. Leave new
Thanks
Thx for sharing and kudos for choosing torrent.
Hi there. For those who want to play with the data on their system, but via the Stack Exchange Data Explorer UI / WebApp and not boring ‘ol SSMS, you can run that locally as well :). Just download the WebApp at:
https://github.com/StackExchange/StackExchange.DataExplorer
So I’m curious…and in my googling around, I couldn’t find what I thought was a good answer. Why is the StackOverflow Database set up as case sensitive? It was actually a nice little learning opportunity for me as I didn’t even know you could make column names etc., case sensitive. Thanks.
Hi Dayton. That would be a good question to ask on: http://meta.stackexchange.com/ (be sure to tag it with [data-explorer] ).
For now, here is a related question from there that doesn’t answer why case-sensitivity was chosen, but how to deal with it (be sure to read the comments as well 🙂
http://meta.stackexchange.com/questions/119304/why-is-the-like-operator-case-sensitive-in-data-explorer
Dayton – it’s a function of the database importer. You set the collation when you create a database. I use case sensitivity on all of my database servers because I want my scripts to work on everyone’s servers, and there’s a surprisingly large number of case-sensitive servers out there. Forcing my stuff to be case-sensitive from the start means I get less support calls on my scripts down the road.
Well that’s a very good answer then Brent. I’m curious though. How do you handle text searching. StackOverflow is super fast. But there is no way it would be that fast if you are UPPERING() or LOWERING() all of the text searches. That’s a quick way to defenestrate the sargability. Although, I have to admit, I’m a bit of a padawan when it comes to SQL. Side note, it would be pretty cool if you could include the actual indexes used (as a script) Thanks!
Dayton – the text search is done in ElasticSearch.
When you say the actual indexes, can you elaborate on what you mean?
Thanks Brent. The actual indexes that are on the production DB. I’m guessing there are more than just a clustered index on each table. Something like a script that has all the create index statements.
Dayton – ah, no, that wouldn’t be appropriate.
Okey Dokey. Thanks for the db/torrent etc.
I don’t understand how then we are to use this with the stress test Random_Q SP if none of the tables have indexes.
Luis – that’s part of your job – to tune them by finding the right indexes during your load tests. (Hey, if I gave you a perfect database with perfect queries and perfect indexes, then you wouldn’t learn anything, hahaha.)
That’s amazing! Thank you, Brent, for sharing this!
Denis – you’re welcome, glad I could help.
Why is this a zipped MDF & LDF and not a BAK file? I would have thought a full database backup would have been a more natural way to distribute the data set than an unattached MDF/LDF.
Wyatt – great question! Because in order to get the file size down, I would still have to compress the backup with 7Z. That means you would have to have enough space for the 7Z, the backup, and the MDF/LDF. The extraction time would also be longer, because it takes you a long time to restore a 70GB database. This way, you need less space (just the 7Z and MDF/LDF), and much less time (extract, then attach).
I’m seeding this now, I’ll let it run as long as the wife doesn’t yell at me for the internet being slow.
Hahaha, thanks!
I have following error during extract.
“Unsupported compression method error for the file…”
Does it mean that I need to download the file once again?
When I’m checking the file, the method is set to 21.
Listing archive: F:\StackOverflow201508.
Method = 21
Solid = –
Blocks = 3
Physical Size = 9240856556
Headers Size = 228
———-
Path = Readme.txt
Size = 1365
Packed Size = 788
Modified = 2015-09-27 16:06:56
Attributes = ….A
CRC = BC21062A
Encrypted = –
Method = 21
Block = 0
Path = StackOverflow.mdf
Size = 69038243840
Packed Size = 9206452306
Modified = 2015-09-27 15:59:26
Attributes = ….A
CRC = 7E857A08
Encrypted = –
Method = 21
Block = 1
Path = StackOverflow_log.ldf
Size = 524165120
Packed Size = 34403234
Modified = 2015-09-27 15:59:26
Attributes = ….A
CRC = 330FDAC2
Encrypted = –
Method = 21
Block = 2
Krzysztof – sorry, I can’t troubleshoot that for you. You may want to try a different extraction tool as well.
Added to my 1 gigabit seedbox. I’ll have it up for at least a month, so it should help anyone who wants to get it.
Thanks sir!
I noticed that the most recent users and posts in your data dump are from September 14th **2014**, not 2015. So this snapshot is over a year old, correct?
There are some missing tables like CloseAsOffTopicReasonTypes, TagSynonyms . Any thoughts?
Jay – they’re not in the public data dump, right?
Helloo,
Does this mean we have the source code of the website ? Or it is juste the database, what can we do with this database ? Thank you
Just the database. If you don’t know what you can do with a database, then this isn’t really for you. Thanks!
I am trying to download the BiTtorrent file to follow you on the training videos, however its been trying to connect to peers over two hours? Can i also get the same file from https://archive.org/details/stackexchange.
James – unfortunately we can’t troubleshoot that remotely. It’s working fine here though.
I have downloaded the file and extracted it. But when I try to attach it I get It asks for full-text catalog to be added but there is no one.
How can I solve this?
Shaui – there isn’t one. You can skip that part.
Thanks.
My mistake.
The real problem was database limitation because the instance was sqlexpress and the limit is 10 gig database.
I am moving the files to another server.
There’s two different torrent files linked in the article. The top one is newer (going by the name).
Thanks for compiling and publishing this DB – it’s a great help for learning!
James – great catch! Fixed. Thanks!
Uh oh, doesn’t look like either of the torrent links works at this time.
Working fine here. Your office may be blocking BitTorrent.
Seems there aren’t enough seeds? FrostWire is telling me the download will be completely in an infinite number of days, hours, minutes and seconds from now!! I’ve managed to get 57.7KB so far…..
completely = completed
Still pointing to the old (201603) torrent
in the download instructions that is
Thanks, fixed.
Thanks Brent Ozar team! This is helpful as always. Do you have any recommendations for free sample databases that are a bit smaller, such as 10-20 GB? I like the Stack dump, but it’s now at the point where it’s surpassing the size of many laptop SSDs, which is where I’d like to mess around with it.
M – you can buy a 1TB laptop SSD for under $250:
http://amzn.to/2l0bMNz
Here is a direct download link: http://ovh.to/D4JmSb6 (hosted on Hubic in France)
WQW – awesome, thanks! I’ve added that to the post.
[…] StackOverflow sample database, shared on BitTorrent by Brent Ozar […]
Is it possible to use the stackoverflow database in SQL Server 2016?
Yep, 2005 and up.
Hey Brent,
We enjoy and learn more with the StackOverFlow DB. Thanks for the great work.. Post downloading we rebuilt the clustered index as Clustered Columstore store and size was reduced by around 30%. Would it possible to add in the future torrents release? which would save us a lot of space..
Kannan – glad you like it. No, not all versions support columnstore indexes.
Why there are only 9 tables? Did I set up the database in a wrong way?
The tables I can see are:
PostTypes
Votes
Users
Posts
Comments
Badges
PostLinks
LinkTypes
VoteTypes
Yep. What other tables were you expecting to see, just curious?
The original database on Data.StackExchange has many more tables. One of them is the dbo.Tags, which is used in one of your first lessons. I wanted to use these lessons to perform live SQL training for my colleagues.
I was expecting to see all of them.
I might reconsider using the AdventureWorks database (or the new MS training db?)
I might use the SE database for some query tuning training or something like that.
Hmmm, which lesson? I don’t remember using an existing dbo.Tags table (although I know in one of my lessons, I have you CREATE one.) Can you point me to it? I want to make sure I get that fixed. Thanks!
Also, to be clear – the Data.StackExchange.com database is a backup of production, whereas the process to create the public data dump (XML) is a little different, and has never contained all of the production tables/columns.
Brent, it’s this lesson:
https://www.brentozar.com/learn-query-sql-server-stackoverflow-database/learn-query-part-1-getting-data-select/from-getting-data-table/
Gotcha. That class is specifically designed for folks using Data.StackExchange.com, not the Stack Overflow data dump. (You’ll notice that the instructions all focus on Data.StackExchange.com.) That’s different from the data dump.
The StackExchange is an excellent DBA “toy”. Thank you for providing an easy way to access it Brent!
One question that I am curious about: how do you import the XML files into your SQL Server?
Jakob – we use the Stack Overflow Data Dump Importer: https://github.com/BrentOzarULTD/soddi
are these datasets contains the question? and tags assigned to them or not?
Yep!
[…] database on your machine to play around, check out Brent Ozar’s instructions on how to do it here). My laptop is running Crucial MX200 1TB SSD. My throughput is 490.491 MB / sec (see snapshot […]
Hi
Thx for this, just downloaded and attached to a SQL Server 2017 instance.
I see that non-clustered-indexes has been removed, but does that also go for foreign key constraints? Or are there none of those in the prod-DB?
Just wanted to mention: when I attached the mdf-file (the first one, there were 3 .mdf-files in the download), it complained if the log-file from the download already was in the log-file-folder. Seems it wanted to create it.
Gert – the data dump isn’t a direct backup of Stack Overflow’s production database. They export the data to XML, and then we import it into SQL Server format. The tables aren’t necessarily identical in structure to Stack’s live schema – it’s very highly similar, but not identical.
Sounds like you didn’t quite follow the process of normal database attachment, but that’s totally okay. Have fun with it!
[…] How to Download the Stack Overflow Database via BitTorrent […]