White paper on Spool space in teradata
4 likes3,971 views
The document discusses Teradata's spool space, which is crucial for query processing and can often lead to errors if not managed properly. It identifies causes of spool space errors, such as inefficient SQL queries and poor indexing, while recommending best practices for capacity planning and query optimization. Key considerations include understanding data distribution, collecting accurate statistics, and choosing appropriate table types to minimize spool space usage.
White paper on Spool space in teradata
- 1. Spool Space in Teradata - Nazir Iqbal 07-March-2012 1
- 2. Table of Contents 1. Introduction………………………………………………………………………………………….3 1.1 Spool Space………………………………………………………………………………….3 1.2 Spool Space and Capacity Planning………………………………………………….3 1.3 Spool Space Categories………………………………………………………………….4 1.4 Spool Space Allocation……………………………………………………………………4 2. Causes of spool space error and how to minimize it……………………………………4 3. Know the data…………………………………………………………………………………………5 4. Primary Index…………………………………………………………………………………………5 5. Multiset or Set table…………………………………………………………………………………6 6. Collect Statistics………………………………………………………………………………………7 7. Skewing………………………………………………………………………………………………….7 8. Conclusion……………………………………………………………………………………………….8 9. References………………………………………………………………………………………………8 2
- 3. 1. INTRODUCTION: 1.1 SPOOL SPACE: TERADATA Spool Space is unused Perm Space that it used for running queries. Spool Space is used to hold intermediate rows during processing, and to hold the rows in the answer set of a transaction. TERADATA recommends 20% of the available perm space is allocated for Spool space but various across applications. In the majority of cases, well written SQL queries should not use huge amounts of spool space. A poor choice of join column, product join and lack of statistics are the main reason of excessive spool space consumption. Each user can be set a spool space limit. In later version of TERADATA, this is often set in the user’s profile. Insufficient spool error is usually the result of poor table design, poor data distribution, or a poorly written query. Running out of Spool Space will give the user an error code 2646. 1.2 SPOOL SPACE AND CAPACITY PLANNING: H E C K Spool Space and Capacity Planning are mutually dependent concepts. Spool space is critical to the operation of Teradata RDBMS, yet it is frequently overlooked in capacity planning. Size requirements vary from user to user, table to table and application to application. For instance, • The Spool space of a user is used to hold the response rows of every query run by that user during a session. Thus, each user needs a high enough spool allocation to contain the biggest anticipated answer set. •Tables containing huge data require more available spool space than smaller tables, because intermediate rows are held in spool space during query execution. 1.3. SPOOL SPACE CATEGORIES Spool falls into three categories of space. They are:- Volatile Intermediate Output 3
- 4. Volatile Intermediate Output This Spool is retained until the Intermediate spool results are Output Spool results are either transaction completes (unless retained until no longer needed. final rows returned in the answer the table was created with ON We can determine when set for a query, rows updated COMMIT PRESERVE ROW), table intermediate spool is flushed by within, inserted into, or deleted is dropped manually during the examining the output of an from a base table. session, Session ends or Teradata EXPLAIN. RDBMS resets. 1.4. SPOOL SPACE ALLOCATION Teradata RDBMS allocates spool space dynamically only from disk cylinders that are not being used for permanent or temporary data. Permanent, temporary, and spool data blocks cannot co-exist on the same cylinder. Spool space is not reserved. All unused space in the Teradata RDBMS is considered available spool space. When spool is released, the file system returns the cylinders it was using to the free cylinder list. We allocate spool space for a database, a user, or a user profile, not at the table level. A SPOOL limit defined in a profile takes effect upon completion of a: • CREATE/MODIFY USER statement that assigns the profile to a user. • MODIFY PROFILE statement that changes the spool space limit. If the user is logged on, the profile specification affects the current session. Inefficient SQL queries generally results in Capacity Planning and Spool Space Allocation going wayward and throws up Spool Space Error which is one of the most common error encountered by a Teradata SQL Programmer. 2. CAUSES OF SPOOL SPACE ERROR AND HOW TO MINIMIZE IT: When Resource thresholds are met, like Spool Space exceeded, then either a warning is given by the DBAs or the query is aborted by them. Different thresholds are set for tactical, decision support, and ad-hoc scenarios. High skew is another cause for Spool Space been exceeded. Not all alerts or warnings indicate there is a problem, as some transactions use high CPU and spool because of large data volumes and complexity of the code. Often we have en-countered scenarios where a SQL query has been running for a long time. The reasons may be: • Missing or aged statistics. • Large product joins. • Merge joins where there is a many-to-many relationship. 4
- 5. • Set tables that should be Multiset. • Stats reflect zero rows on a table, yet are not empty. • A change in data volume which requires additional stats and it will generate a different explain plan. • Unbalanced parenthesis. The key is to know the data before writing SQL codes. 3. KNOW THE DATA Below are a few questions that should always be given a thought so that SQL codes are efficient and do not exceed the thresholds of Spool Space or CPU. 1. How many rows exist on the tables in the query? 2. What columns are we joining on? 3. Do we need to add filters or additional joins to reduce volume? 4. How many unique values exist on columns? 5. How many rows exist on tables that are duplicated? 6. Queries having derived tables will often show no confidence because the optimizer does not know how many rows are in a derived table. 7. High estimated time can indicate aged stats i.e. stats should be collected again. 8. What type of join is performed? Product Join Merge Join Hash Join This is a cross join every row from Requires sort of spool files. Merge The tables do not have to be sorted one table is joined to every row on join are efficient when there is not and the smaller table can be much the second table. Spool file is as a many to many relationship on larger than for a product join. The large as (No. Of rows table_one * columns involved in the join. smaller table/spool is "hashed" into No. Of rows table_two), large If there is a many to many memory. Then, the larger table is product joins (billions of rows) relationship, try to aggregate the scanned and for each row, it looks should be avoided. Product joins columns on one table to reduce the up the row from the smaller table are most efficient when a SMALL volume by creating a volatile table, in the hashed table that was lookup table is duplicated. Product derived table or work table. created in memory. If the smaller joins are inefficient when large fact table is broken into partitions to fit tables are duplicated (this can into memory, the larger table must indicate aged or missing stats). also be broken into the same partitions prior to the join. 4. PRIMARY INDEX A poor primary index having lumpy distribution data which can cause a query to run several hours when it should execute in seconds/minutes. Hence, we should choose a single column or multiple columns that distribute the data evenly across all AMPS. 5
- 6. Eliminate columns from the primary index that have a lot of null values. Value change rate should be low or never. Column(s) should be frequently used in join constraint. Teradata is a multi parallel processor so a query runs as well as the SLOWEST AMP. If the table joins to a similar table having the same columns, the primary index on both tables should be the same. AMP-4 has much more data than AMP-1, AMP-2 and AMP-3 which causes Spool Space Error. Choice of primary index should be such to avoid such un-even data distribution across AMPs. 5. MULTISET OR SET TABLE A set table performs a duplicate row check. If there are a lot of non unique values for a primary index, this can be very CPU intensive. For example, for a primary index having 2000 values a duplicate row check will be performed 4,000,000 times. This is referred to as chaining. The first record is loaded. The next record having the same PI value to load, checks all the columns of the first one to determine if it is a duplicate. Once the third record is loaded, it checks both the first and the second records and so on. A Multiset table allows duplicate rows so the duplicate row check is omitted. If duplicates can be omitted using a group by or filtered programmatically, a load to a multiset table performs better. A Multiset table having a NUPI, non unique primary index, with occurrences between 500 – 2000 is not bad. For tables having non unique primary index where there are several hundred or a couple of thousands values for a given primary index ‘use a multiset table’ For tables having a more unique index like 1 to 10 values for a give primary index ‘use a set table’ Note: the FASTLOAD utility program will not allow duplicates, even if the target table is MULTISET. 6
- 7. 6. COLLECT STATISTICS Poor or missing statistics OR Aged statistics may cause Spool Space Error. TERADATA recommends that COLLECT STATS should include:- 1. Individual columns in an index. 2. All columns in an index, multi-column where size is less than 16 bytes. 3. Join columns. 4. Filter or qualifying columns. 5. Secondary Index Statistics are not needed for temp tables that are not joined to other tables and only used for staging. Be careful to NOT over collect on statistics. If a table is updated by several inserts multiple times a day, the statistics do not need to be refreshed after each insert. One collection is Significant after the last insert. For tables being completely refreshed, the statistics are Needed after the refresh. TOOL TO CHECK THE EFFICIENCY OF A QUERY: Run this diagnostic command before the explain of the query. At the bottom of the explain it will list the statistics that are missing. Diagnostic helpstats on for session; Explain <OUR SQL TEXT> WE SHOULD NOT COLLECT STATISTICS ON EMPTY TABLES. This will cause the optimizer to choose an inefficient path based on the information available to the parser. Statistics should be collected when a table is initially loaded and anytime the table’s demographics change by more than 10%. After the initial collect statistics on an object, the user can run the statement below to refresh the table’s statistics based on the (new) data. Collect statistics on databasename.tablename; -- this will refresh all stats, index and column, that were previously gathered on a table To see the statistics that exist for a table, run the following: HELP STATISTICS databasename.tablename; 7. SKEWING: Proper primary index specification should evenly distribute the rows of a table across the AMPs. This prevents skewing. The Query Log Information in SQL Assistant and other Editors tells us about the degree of Skewing in a query.CPU Skew > 50 reflects worse case scenarios and generally any query having a CPU Skew > 4 is considered poor performing. Hence, by seeing the CPU Skew from the Query Log Information a programmer can easily make out which query needs to be fine-tuned to avoid high Skew. 7
- 8. CONCLUSION: Most of the performance related issues are caused by poor indexing, missing statistics, aged statistics , over collecting statistics, mismatched data types and missing filters and conditions on where clause. These can be eliminated if we follow the best practices of sound SQL techniques as discussed above. The key to efficient SQL coding is good knowledge of the database and understanding the various join-constraints and the mappings. Database knowledge accompanied with adherence to collect stats feature of TERADATA is the key to avoidance of Spool Space Error. REFERENCES: 1. http://www.info.teradata.com/Datawarehouse/eBrowseBy.cfm?page=TeradataDatabase 2. http://www.teradatatech.com/ 3. www.google.com 8