Why product development benefits from clean data structures
One of the websites I am currently dealing with is pretty data based. Its development so far though has not been data driven. All the data was pretty much just a byproduct which we will now try to monetize by developing new products from it.
For that the data has to be clean structured.
The website I am referring to is tightly coupled to a print magazine. Whenever a new issue is out the website needs to be updated.
All data that was gathered during the research for the new issue is put into the database and the website has to produce some reports that resemble the ones from the magazine.
Of course a print magazine has no data structure as such just a bunch of visualizations which change with every issue. The data as well as the reports are pretty volatile. This way every issue is interesting again and not just the same as before but with new data.
For the website this means that the quality, the quantity and the granularity of the data changes a lot which means a lot of changing of the data structures. It’s easy to imagine huge char or even blob fields that hold json or xml. Development like this can lead to pretty ugly solutions.
Basically we end up in a situation where we do not know which data we have and for what time frame. We don’t know if it is able to be queried for or if we can separate some figures from others. We can look it up of course but not while developing new product ideas.
If we would have a stable data structure that is optimized for flexible querying in order to cut very different segments from it we could easily assemble data for various contexts that we could sell or otherwise monetize.
To get from the actual to the target situation we need to find out which part of our data is stable and always present historically and which parts vary a lot.
Next up we would probably need to define which part of our data is best kept in what kind of data store. Some part will be ideally stored in a relational database, other maybe in a document based database or key value store of graph db.
Once we reach that target and we’re able to maintain this structure we will be able to spin off byproducts as soon as we think of them. Otherwise we’re stuck.