Database tools for technologists - short

Editor's Notes

• #3: Technologists generate and deal with data of many different types, in multiple formats, and in volumes that can be small, medium, large,or enormous. Given that technologists want to benefit from (or at least explore the feasibility of) working with a database management system (a DBMS), the issue at hand is - how well can a DBMS be adapted to technologists’ needs? Our database extensions (or plug-ins), which install into popular object relational DBMSs, speed up data loading and querying, shrink data space requirements – and hence reduce storage costs. In addition, they shield users from much computational and data management complexity.
• #4: The three BCS database extension products presented here deal with storing and querying gridded data, storing and querying large data series, and providing SQL querying facilities over top of traditional files.
• #5: I’ll talk about the Grid DataBlade first. We developed the Grid DataBlade to address specific needs of the US Navy to customize and speed up its generation and delivery of tactical weather forecasts to the fleet. FNMOC, in Monterey, California,provides worldwide weather information to the US Navy. It generates very large 4D grids each covering a vast expanse of the Globe, but what’s of interest to a pilot on a mission are the forecast conditions around his or her plane, for the duration of that mission. So the Grid DataBlade is used to extract that relatively small amount of vital information from the large grid. The consequent productivity gains experienced at FNMOC from using the Grid DataBlade exceeded an order of magnitude.
• #6: Our second product is DBXten. The motivation for DBXten was to address the issue of ingesting into a database fast streams of data sequences – that is, complex data, produced, say, by multiple sensors, related to any of the sources shown in this picture.
• #7: There is extensive information about DBXten on our website, and we’ve done lots of performance testing, also documented on our website. This slide shows the impressive performance results we achieved in one of these tests. I’ll leave this slide up for a few more seconds as we’re quite proud of these results – which are typical of the gains achieved with DBXten! BCS holds US Patent 8077059 for DBXten.
• #8: Our latest product, the Universal File Interface (UFI), is designed to address one of the primary reasons we’ve heard from technologists who avoid, or limit their use of, DBMSs “…my files are too cumbersome/complex/large to load into a database”. UFI allows users to index and query their files without having to actually load them into a database! Files often have an inherent structure: CSV files have columns, separated by commas, DBF files have fields, NetCDF files have attributes and columns, and so on. With UFI we map these file components to columns of virtual tables. If we want, we can map several files to a single table, thereby creating a virtual table that represents the union of the data in many files. And we can perform SQL on that virtual table, just as if it were a real table in a database. The only difference is – we didn’t have to spend time loading it.
• #9: UFI is one of the products behind a popular demo on our website – we call it the “Weatherdemo”, shown in this picture.
• #10: What it allows you to do is zoom and pan around the contiguous United States and look at the forecasts made for a particular day and time in the future, and then compare those forecasts with the actual weather recorded on that day and time. So, every day we download gridded binary files containing temperature, wind, and rain forecast data for the USA; we then use UFI and SQL to join this forecast data, stored in files, with actual weather observed values, stored in a database. Our website explains what information is available in each colored square, circle, teardrop, and little star depicted above (e.g., clicking on a teardrop will give you a textual display of how the forecast compared with the actual conditions at that location and time).
• #11: This concludes our mini presentation on Database Tools for Technologists. Thank you for your interest. For more information please visit our website, send us an email, or give us a call. The final bullet above points to an in-depth down-loadable presentation on Database Tools for Technologists. Goodbye for now.