Abstract image of a woman sitting on books and studying on a laptop

Writing A Foreign Data Wrapper

P
psoo1978

The document details the creation and implementation of a foreign data wrapper (FDW) for PostgreSQL, specifically focusing on the Informix database. It outlines necessary steps, functions, and callback routines involved in accessing and integrating remote data sources as PostgreSQL tables. Key topics include error handling, data conversion, and the importance of ensuring encoding compatibility.

Technology
1 of 46
Downloaded 27 times
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
Writing A Foreign Data Wrapper Bernd Helmle, bernd.helmle@credativ.de 30. Oktober 2013
Why FDWs? ...it is in the SQL Standard (SQL/MED) ...migration ...heterogeneous infrastructure ...integration of remote (non-)relational datasources ...it’s cool http: //rhaas.blogspot.com/2011/01/why-sqlmed-is-cool.html
Access remote datasources as PostgreSQL tables... CREATE EXTENSION IF NOT EXISTS informix_fdw; CREATE SERVER sles11_tcp FOREIGN DATA WRAPPER informix_fdw OPTIONS ( informixdir ’/Applications/IBM/informix’, informixserver ’ol_informix1170’ ); CREATE USER MAPPING FOR bernd SERVER sles11_tcp OPTIONS ( password ’informix’, username ’informix’ ); CREATE FOREIGN TABLE bar ( id integer, value text ) SERVER sles11_tcp OPTIONS ( client_locale ’en_US.utf8’, database ’test’, db_locale ’en_US.819’, query ’SELECT * FROM bar’ ); SELECT * FROM bar;
What we need... ...a C-interface to our remote datasource ...knowledge about PostgreSQL’s FDW API ...an idea how we deal with errors ...how remote data can be mapped to PostgreSQL datatypes ...time and steadiness Python-Gurus also could use http://multicorn.org/.
Before you start your own... Have a look at http://wiki.postgresql.org/wiki/Foreign_data_wrappers
Let’s start... extern Datum ifx_fdw_handler(PG_FUNCTION_ARGS); extern Datum ifx_fdw_validator(PG_FUNCTION_ARGS); CREATE FUNCTION ifx_fdw_handler() RETURNS fdw_handler AS ’MODULE_PATHNAME’ LANGUAGE C STRICT; CREATE FUNCTION ifx_fdw_validator(text[], oid) RETURNS void AS ’MODULE_PATHNAME’ LANGUAGE C STRICT; CREATE FOREIGN DATA WRAPPER informix_fdw HANDLER ifx_fdw_handler VALIDATOR ifx_fdw_validator;
FDW handler Creates and initializes a FdwRoutine structure, example: Datum ifx_fdw_handler(PG_FUNCTION_ARGS) { FdwRoutine *fdwRoutine = makeNode(FdwRoutine); fdwRoutine->ExplainForeignScan = ifxExplainForeignScan; fdwRoutine->BeginForeignScan = ifxBeginForeignScan; fdwRoutine->IterateForeignScan = ifxIterateForeignScan; fdwRoutine->EndForeignScan = ifxEndForeignScan; fdwRoutine->ReScanForeignScan = ifxReScanForeignScan; #if PG_VERSION_NUM < 90200 fdwRoutine->PlanForeignScan = ifxPlanForeignScan; #else fdwRoutine->GetForeignRelSize = ifxGetForeignRelSize; fdwRoutine->GetForeignPaths = ifxGetForeignPaths; fdwRoutine->GetForeignPlan = ifxGetForeignPlan; #endif
FDW handler ...and much more callbacks for DML with PostgreSQL 9.3 Datum ifx_fdw_handler(PG_FUNCTION_ARGS) { [...] #if PG_VERSION_NUM >= 90300 fdwRoutine->AddForeignUpdateTargets fdwRoutine->PlanForeignModify fdwRoutine->BeginForeignModify fdwRoutine->ExecForeignInsert fdwRoutine->ExecForeignDelete fdwRoutine->ExecForeignUpdate fdwRoutine->EndForeignModify fdwRoutine->IsForeignRelUpdatable #endif PG_RETURN_POINTER(fdwRoutine); } = = = = = = = = ifxAddForeignUpdateTargets; ifxPlanForeignModify; ifxBeginForeignModify; ifxExecForeignInsert; ifxExecForeignDelete; ifxExecForeignUpdate; ifxEndForeignModify; ifxIsForeignRelUpdatable;
FDW validator callback Called via CREATE FOREIGN TABLE or ALTER FOREIGN TABLE Validates a List * of FDW options. Use untransformRelOptions() to get a list of FDW options Don’t forget to test for duplicated options! Up to you which options you want to support Have a look into foreign/foreign.h for various helper functions
FDW API callback routines (1) #ifdef PG_VERSION_NUM < 90200 static FdwPlan *PlanForeignScan(Oid foreignTableOid, PlannerInfo *planInfo, RelOptInfo *baserel); #else static void GetForeignRelSize(PlannerInfo *root, RelOptInfo *baserel, Oid foreignTableId); static void GetForeignPaths(PlannerInfo *root, RelOptInfo *baserel, Oid foreignTableId); static ForeignScan *GetForeignPlan(PlannerInfo *root, RelOptInfo *baserel, Oid foreignTableId, ForeignPath *best_path, List *tlist, List *scan_clauses); #endif
FDW API callback routines (2) static void ExplainForeignScan(ForeignScanState *node, ExplainState *es); static void BeginForeignScan(ForeignScanState *node, int eflags); static TupleTableSlot *IterateForeignScan(ForeignScanState *node); static void EndForeignScan(ForeignScanState *node);
FDW API callback routines (3) 9.2 has callbacks for ANALYZE, too: bool AnalyzeForeignTable (Relation relation, AcquireSampleRowsFunc *func, BlockNumber *totalpages); int AcquireSampleRowsFunc (Relation relation, int elevel, HeapTuple *rows, int targrows, double *totalrows, double *totaldeadrows);
FDW API callback routines (4) 9.3 introduces callbacks for DML actions #if PG_VERSION_NUM >= 90300 fdwRoutine->AddForeignUpdateTargets fdwRoutine->PlanForeignModify fdwRoutine->BeginForeignModify fdwRoutine->ExecForeignInsert fdwRoutine->ExecForeignDelete fdwRoutine->ExecForeignUpdate fdwRoutine->EndForeignModify fdwRoutine->IsForeignRelUpdatable fdwRoutine->ExplainForeignModify #endif = = = = = = = = = ifxAddForeignUpdateTargets; ifxPlanForeignModify; ifxBeginForeignModify; ifxExecForeignInsert; ifxExecForeignDelete; ifxExecForeignUpdate; ifxEndForeignModify; ifxIsForeignRelUpdatable; ifxExplainForeignModify;
FDW Flow AddForeignTargets() GetForeignRelSize() GetForeignPaths() Query Cost Estimates Query Planning Query Planning - Finalize Prepare / Metadata GetForeignPlan() PlanForeignModify() IsForeignRelUpdatable() BeginForeignScan() BeginForeignModify() Materialize IterateForeignScan() ExecForeignUpdate() ExecForeignDelete() ExecForeignInsert() EndForeignScan() Free resources
AddForeignTargets static void ifxAddForeignUpdateTargets(Query *parsetree, RangeTblEntry *target_rte, Relation target_relation); Injects a custom column into the parsetree for DELETE and UPDATE commands. Can be NULL in case DELETE is not supported or UPDATE actions rely on unchanged unique keys.
FDW Query Planning Setup and Planning a scan or modify action on a foreign datasource E.g. establish and cache remote connection Initialize required supporting structures for remote access Planner info and cost estimates via baserel and root parameters. Big differences between 9.1 and 9.2 API, DML introduced with 9.3 API
GetForeignRelSize() (1) Size estimates for remote datasource (table size, number of rows, ...) root: Query Information Structure baserel: Table Information Structure, carry your FDW private information in baserel->fdw private. baserel->fdw private is a pointer to your state data structure.
GetForeignRelSize() (2) baserel->rows = ifxGetEstimatedNRows(&coninfo); baserel->width = ifxGetEstimatedRowSize(&coninfo); baserel->fdw_private = (void *) planState;
GetForeignPaths() (1) Create access path for foreign datasource. ForeignPath access path required at least. Multiple paths possible (e.g. presorted results, ...) Arbitrarily complex
GetForeignPaths() (2) planState = (IfxFdwPlanState *) baserel->fdw_private; /* * Create a generic foreign path for now. We need to consider any * restriction quals later, to get a smarter path generation here. * * For example, it is quite interesting to consider any index scans * or sorted output on the remote side and reflect it in the * choosen paths (helps nested loops et al.). */ add_path(baserel, (Path *) create_foreignscan_path(root, baserel, baserel->rows, planState->coninfo->planData.costs, planState->coninfo->planData.costs, NIL, NULL, NIL));
GetForeignPlan() (1) Creates a final ForeignScan plan node based on paths created by GetForeignPaths() Additional parameters ForeignPath *best path: Choosen foreign access path (best) List *tlist: Target list List *scan clauses: Restriction clauses enforced by the plan
GetForeignPlan() (2) ForeignScan plan node should be created by make foreignscan(): ForeignScan * make_foreignscan(List *qptlist, List *qpqual, Index scanrelid, List *fdw_exprs, List *fdw_private) fdw exprs: Expressions to be evaluated by the planner fdw private: Private FDW data
Passing FDW planning info to execution state Save parameters in the foreignScan->fdw_private pointer. Must be copieable with copyObject. Use a List * with either bytea or/and constant values (via makeConst). List *plan_values; plan_values = NIL; plan_values = lappend(plan_values, makeConst(BYTEAOID, -1, InvalidOid, -1, PointerGetDatum(ifxFdwPlanDataAsBytea(coninfo)) false, false));
Predicate Pushdown Challenge: Filter the data on the remote dataset before transferring them Nobody wants to filter thousands of rows to just get one Idea: push filter conditions down to the foreign datasource (if possible) Ideally done during planning phase (GetForeignRelSize(), GetForeignPaths()) baserel->baserestrictinfo Hard to get it right
Predicate Pushdown baserel->baserestrictinfo: List of predicates belonging to the foreign table (logically AND’ed) baserel->reltargetlist: List of columns belonging to the foreign table Have a look at expression tree walker() and ruleutils API (include/nodes/nodeFuncs.h, include/utils/ruleutils.h) ListCell *cell; foreach(cell, baserel->baserestrictinfo) { RestrictInfo *info; info = (RestrictInfo *) lfirst(cell); if (IsA(info->clause, OpExpr)) { /* examine right and left operand */ } }
Predicate Pushdown, Example SELECT COUNT(*) FROM sles11.inttest; count ------10001 (1 row) EXPLAIN SELECT * FROM foo JOIN centosifx.inttest t ON (t.f1 = foo.id) WHERE t.f1 = 104 AND t.f2 = 120; QUERY PLAN --------------------------------------------------------------------------------Nested Loop (cost=2.00..1695.03 rows=1 width=18) -> Seq Scan on foo (cost=0.00..1693.01 rows=1 width=4) Filter: (id = 104) -> Foreign Scan on inttest t (cost=2.00..2.01 rows=1 width=14) Filter: ((f1 = 104) AND (f2 = 120)) Informix costs: 2.00 Informix query: SELECT * FROM inttest WHERE (f1 = 104) AND (f2 = 120) (7 rows)
PlanForeignModify Create modify state information, e.g. get a connection, describe remote metadata, ... Private state information attached to ModifyTable plan node State information pushed down to BeginForeignModify Same with planning a scan: private state information must be in a List * format. INSERT probably needs more work here (connection preparing, ...)
PlanForeignModify - Access scan state Possible to access the scan state created during planning phase static List * ifxPlanForeignModify(PlannerInfo *root, ModifyTable *plan, Index resultRelation, int subplan_index) { if ((resultRelation < root->simple_rel_array_size) && (root->simple_rel_array[resultRelation] != NULL)) { RelOptInfo *relInfo = root->simple_rel_array[resultRelation]; IfxFdwExecutionState *scan_state; /* * Extract the state of the foreign scan. */ scan_state = (IfxFdwExecutionState *) ((IfxFdwPlanState *)relInfo->fdw_private)->state; } }
BeginForeignScan() void BeginForeignScan (ForeignScanState *node, int eflags); Execute startup callback for the FDW. Basically prepares the FDW for executing a scan. ForeignScanState saves function state values. Use node->fdw_state to assign your own FDW state structure. Must handle EXPLAIN and EXPLAIN ANALYZE by checking eflags & EXEC_FLAG_EXPLAIN_ONLY
BeginForeignModify Like BeginForeignScan(), starting callback for DML actions into the executor. Depending on the DML action, might be differences in preparing stuff E.g. setup connection or get connection from cache Deserialize plan data pushed down from PlanForeignModify()
IterateForeignScan() (1) TupleTableSlot * IterateForeignScan (ForeignScanState *node); Fetches data from the remote source. Data conversion Materializes a physical or virtual tuple to be returned. Needs to return an empty tuple when done. Private FDW data located in node->fdw state
IterateForeignScan() (2) Returning a virtual tuple TupleTableSlot *slot = node->ss.ss_ScanTupleSlot; slot->tts_isempty = false; slot->tts_nvalid = number_cols;; slot->tts_values = (Datum *)palloc(sizeof(Datum) * slot->tts_nvalid); slot->tts_isnull = (bool *)palloc(sizeof(bool) * slot->tts_nvalid); for (i = 0; j < attrCount - 1; i) { tupleSlot->tts_isnull[i] = false; tupleSlot->tts_values[i] = PointerGetDatum(val); }
ReScanForeignScan() void ReScanForeignScan (ForeignScanState *node); Prepares the FDW to handle a rescan Begins the scan from the beginning, e.g when used in scrollable cursors Must take care for changed query parameters! Better to just “instruct” IterateForeignScan() to do the right thing (tm)
Execute a modify action(1) static TupleTableSlot * ifxExecForeignUpdate(EState *estate, ResultRelInfo *rinfo, TupleTableSlot *slot, TupleTableSlot *planSlot) static TupleTableSlot * ifxExecForeignInsert(EState *estate, ResultRelInfo *rinfo, TupleTableSlot *slot, TupleTableSlot *planSlot); static TupleTableSlot * ifxExecForeignDelete(EState *estate, ResultRelInfo *rinfo, TupleTableSlot *slot, TupleTableSlot *planSlot);
Execute a modify action (2) ExecForeignUpdate, ExecForeignDelete and ExecForeignInsert callbacks Called for every tuple to work on An INSERT action doesn’t employ a remote scan! Does all the necessary data conversion if necessary slot carries all information for the target tuple Private FDW data stored in rinfo->ri FdwState
EndForeignScan(), EndForeignModify() void EndForeignScan (ForeignScanState *node); void EndForeignModify(EState *estate, ResultRelInfo *rinfo) EndForeignModify() run when no more tuple to modify EndForeignScan() run when IterateForeignScan returns no more rows, but after EndForeignModify() Finalizes the remote scan and modify action Close result sets, handles, connection, free memory, etc...
Transaction Callbacks Connect local transactions with remote transactions and savepoints RegisterXactCallback(pgfdw_xact_callback, NULL); RegisterSubXactCallback(pgfdw_subxact_callback, NULL); static void fdw_xact_callback(XactEvent event, void *arg); static void fdw_subxact_callback(SubXactEvent event, SubTransactionId mySubid, SubTransactionId parentSubid, void *arg)
Memory Management PostgreSQL uses palloc() Memory is allocated in CurrentMemoryContext Use your own MemoryContext where necessary (e.g. IterateForeignScan()) Memory allocated in external libraries need special care
Data conversion Easy, if the remote datasource delivers a well formatted value string (e.g. date strings formatted as yyyy-mm-dd). Use type input function directly Binary compatible types (e.g integer) Binary data should always be bytea String data must have a valid encoding!
Data conversion - Encoding Within a FDW, a backend acts like any other client: ensure encoding compatibility or encode your string data properly. A look at mb/pg wchar.h might be of interest. GetDatabaseEncoding() pg_do_encoding_conversion()
Data conversion - Get type input function regproc typinputfunc; Datum result; HeapTuple type_tuple; type_tuple = SearchSysCache1(TYPEOID, inputOid); if (!HeapTupleIsValid(type_tuple)) { /* * Oops, this is not expected... */ ... } ReleaseSysCache(type_tuple); typinputfunc = ((Form_pg_type) GETSTRUCT(type_tuple))->typinput; result = OidFunctionCall2(typinputfunc, CStringGetDatum(buf), ObjectIdGetDatum(InvalidOid));
Error Handling (1) Set FDW SQLSTATE according to your error condition class HV, see http: // www. postgresql. org/ docs/ 9. 3/ static/ errcodes-appendix. html Alternative: map remote error conditions to PostgreSQL errors Be careful with elog(ERROR, ...).
Error Handling (2) Example (there is no FDW WARNING SQLSTATE): if (err == IFX_CONNECTION_WARN) { IfxSqlStateMessage message; ifxGetSqlStateMessage(1, &message); ereport(WARNING, (errcode(WARNING), errmsg("opened informix connection with warnings"), errdetail("informix SQLSTATE %s: "%s"", message.sqlstate, message.text))); }
Catching Errors (1) Sometimes necessary to catch backend errors Synchronize error conditions between PostgreSQL and remote datasource Possibility: use a PG TRY...PG CATCH block.
Catching Errors (2) PG_TRY(); { ... typinputfunc = getTypeInputFunction(state, PG_ATTRTYPE_P(state, attnum)); result = OidFunctionCall2(typinputfunc, CStringGetDatum(pstrdup(buf)), ObjectIdGetDatum(InvalidOid)); } PG_CATCH(); { ifxRewindCallstack(&(state->stmt_info)); PG_RE_THROW(); } PG_END_TRY();
Thank You!

More Related Content

PPT
15 Ways to Kill Your Mysql Application Performance
guest9912e5
 
PDF
MySQL Query Tuning for the Squeemish -- Fossetcon Orlando Sep 2014
Dave Stokes
 
PDF
ROracle
Mohamed Magdy
 
PDF
Gruter_TECHDAY_2014_04_TajoCloudHandsOn (in Korean)
Gruter
 
PPT
Leveraging Hadoop in your PostgreSQL Environment
Jim Mlodgenski
 
PDF
PGConf APAC 2018 - Lightening Talk #2 - Centralizing Authorization in PostgreSQL
PGConf APAC
 
PPTX
Using Spark to Load Oracle Data into Cassandra
Jim Hatcher
 
PDF
Hypertable Distilled by edydkim.github.com
Edward D. Kim
 
15 Ways to Kill Your Mysql Application Performance
guest9912e5
 
MySQL Query Tuning for the Squeemish -- Fossetcon Orlando Sep 2014
Dave Stokes
 
ROracle
Mohamed Magdy
 
Gruter_TECHDAY_2014_04_TajoCloudHandsOn (in Korean)
Gruter
 
Leveraging Hadoop in your PostgreSQL Environment
Jim Mlodgenski
 
PGConf APAC 2018 - Lightening Talk #2 - Centralizing Authorization in PostgreSQL
PGConf APAC
 
Using Spark to Load Oracle Data into Cassandra
Jim Hatcher
 
Hypertable Distilled by edydkim.github.com
Edward D. Kim
 

What's hot (20)

PPTX
H cat berlinbuzzwords2012
Hortonworks
 
PDF
OSDC 2012 | Scaling with MongoDB by Ross Lawley
NETWAYS
 
ODP
Postgresql Federation
Jim Mlodgenski
 
PDF
Intro To Couch Db
Shahar Evron
 
PDF
Mongodb replication
PoguttuezhiniVP
 
PDF
Zend Server Data Caching
El Taller Web
 
PPTX
Hive data migration (export/import)
Bopyo Hong
 
PDF
Hypertable
betaisao
 
PPT
Database Architectures and Hypertable
hypertable
 
PDF
Hypertable - massively scalable nosql database
bigdatagurus_meetup
 
PDF
Tiered storage intro. By Robert Hodges, Altinity CEO
Altinity Ltd
 
PPTX
Hadoop 20111117
exsuns
 
PPT
July 2010 Triangle Hadoop Users Group - Chad Vawter Slides
ryancox
 
PDF
Introduction to Hadoop
Ovidiu Dimulescu
 
PPTX
Rapid Development of Data Generators Using Meta Generators in PDGF
Tilmann Rabl
 
PDF
Data Exploration with Apache Drill: Day 1
Charles Givre
 
PPTX
Hive commands
Ganesh Sanap
 
PDF
[Pgday.Seoul 2017] 3. PostgreSQL WAL Buffers, Clog Buffers Deep Dive - 이근오
PgDay.Seoul
 
PPTX
Replication and replica sets
Randall Hunt
 
PDF
Jvm goes big_data_sfjava
srisatish ambati
 
H cat berlinbuzzwords2012
Hortonworks
 
OSDC 2012 | Scaling with MongoDB by Ross Lawley
NETWAYS
 
Postgresql Federation
Jim Mlodgenski
 
Intro To Couch Db
Shahar Evron
 
Mongodb replication
PoguttuezhiniVP
 
Zend Server Data Caching
El Taller Web
 
Hive data migration (export/import)
Bopyo Hong
 
Hypertable
betaisao
 
Database Architectures and Hypertable
hypertable
 
Hypertable - massively scalable nosql database
bigdatagurus_meetup
 
Tiered storage intro. By Robert Hodges, Altinity CEO
Altinity Ltd
 
Hadoop 20111117
exsuns
 
July 2010 Triangle Hadoop Users Group - Chad Vawter Slides
ryancox
 
Introduction to Hadoop
Ovidiu Dimulescu
 
Rapid Development of Data Generators Using Meta Generators in PDGF
Tilmann Rabl
 
Data Exploration with Apache Drill: Day 1
Charles Givre
 
Hive commands
Ganesh Sanap
 
[Pgday.Seoul 2017] 3. PostgreSQL WAL Buffers, Clog Buffers Deep Dive - 이근오
PgDay.Seoul
 
Replication and replica sets
Randall Hunt
 
Jvm goes big_data_sfjava
srisatish ambati
 
Ad

Viewers also liked (13)

PDF
Foreign Data Wrappers and You with Postgres
EDB
 
PDF
EnterpriseDB Positioned in Leaders Quadrant in Gartner Magic Quadrant
EDB
 
PPTX
NoSQL on ACID: Meet Unstructured Postgres
EDB
 
PDF
Postgres NoSQL - Delivering Apps Faster
EDB
 
PDF
The Security-Enhanced PostgreSQL - "system wide" consistency in access control
Kohei KaiGai
 
PDF
Kevin Kempter PostgreSQL Backup and Recovery Methods @ Postgres Open
PostgresOpen
 
PPTX
PGEncryption_Tutorial
Vibhor Kumar
 
PDF
PostgresOpen 2013 A Comparison of PostgreSQL Encryption Options
Faisal Akber
 
PDF
Postgres.foreign.data.wrappers.2015
EDB
 
PPT
The NoSQL Way in Postgres
EDB
 
PDF
Achieving Pci Compliace
Denish Patel
 
PDF
Security Best Practices for your Postgres Deployment
PGConf APAC
 
PDF
Backup recovery with PostgreSQL
Federico Campoli
 
Foreign Data Wrappers and You with Postgres
EDB
 
EnterpriseDB Positioned in Leaders Quadrant in Gartner Magic Quadrant
EDB
 
NoSQL on ACID: Meet Unstructured Postgres
EDB
 
Postgres NoSQL - Delivering Apps Faster
EDB
 
The Security-Enhanced PostgreSQL - "system wide" consistency in access control
Kohei KaiGai
 
Kevin Kempter PostgreSQL Backup and Recovery Methods @ Postgres Open
PostgresOpen
 
PGEncryption_Tutorial
Vibhor Kumar
 
PostgresOpen 2013 A Comparison of PostgreSQL Encryption Options
Faisal Akber
 
Postgres.foreign.data.wrappers.2015
EDB
 
The NoSQL Way in Postgres
EDB
 
Achieving Pci Compliace
Denish Patel
 
Security Best Practices for your Postgres Deployment
PGConf APAC
 
Backup recovery with PostgreSQL
Federico Campoli
 
Ad

Similar to Writing A Foreign Data Wrapper (20)

PPTX
Accessing external hadoop data sources using pivotal e xtension framework (px...
Sameer Tiwari
 
PDF
Hachiojipm11
Hideaki Ohno
 
PDF
Implementing a basic directory-tree structure that is derived from a.pdf
funkybabyindia
 
PPT
Working with databases in Perl
Laurent Dami
 
PPT
Quebec pdo
Valentine Dianov
 
PDF
Quebec pdo
Rengga Aditya
 
KEY
Operation Oriented Web Applications / Yokohama pm7
Masahiro Nagano
 
PDF
Drupal 7 database api
Andrii Podanenko
 
ODP
Sysprog17
Ahmed Mekkawy
 
PPTX
Apache Hive
Ajit Koti
 
PPTX
UNIT I LINEAR DATA STRUCTURES – LIST .pptx
kncetaruna
 
KEY
Fatc
Wade Arnold
 
PPTX
Sql user group
Stefan Bauer
 
PDF
02 20180605 meetup_fdw_v1
Frederic Bamiere
 
ODP
Zend Framework 1.9 Setup & Using Zend_Tool
Gordon Forsythe
 
PPT
TopicMapReduceComet log analysis by using splunk
akashkale0756
 
PPTX
UNIT I LINEAR DATA STRUCTURES – LIST .pptx
VISWANATHAN R V
 
PDF
Trivadis TechEvent 2016 Polybase challenges Hive relational access to non-rel...
Trivadis
 
PDF
Getting Started with PL/Proxy
Peter Eisentraut
 
PDF
Extbase and Beyond
Jochen Rau
 
Accessing external hadoop data sources using pivotal e xtension framework (px...
Sameer Tiwari
 
Hachiojipm11
Hideaki Ohno
 
Implementing a basic directory-tree structure that is derived from a.pdf
funkybabyindia
 
Working with databases in Perl
Laurent Dami
 
Quebec pdo
Valentine Dianov
 
Quebec pdo
Rengga Aditya
 
Operation Oriented Web Applications / Yokohama pm7
Masahiro Nagano
 
Drupal 7 database api
Andrii Podanenko
 
Sysprog17
Ahmed Mekkawy
 
Apache Hive
Ajit Koti
 
UNIT I LINEAR DATA STRUCTURES – LIST .pptx
kncetaruna
 
Fatc
Wade Arnold
 
Sql user group
Stefan Bauer
 
02 20180605 meetup_fdw_v1
Frederic Bamiere
 
Zend Framework 1.9 Setup & Using Zend_Tool
Gordon Forsythe
 
TopicMapReduceComet log analysis by using splunk
akashkale0756
 
UNIT I LINEAR DATA STRUCTURES – LIST .pptx
VISWANATHAN R V
 
Trivadis TechEvent 2016 Polybase challenges Hive relational access to non-rel...
Trivadis
 
Getting Started with PL/Proxy
Peter Eisentraut
 
Extbase and Beyond
Jochen Rau
 

Recently uploaded (20)

PDF
Taming the Chaos: How to Turn Unstructured Data into Decisions
Safe Software
 
PPTX
Modernising the Digital Integration Hub
Daniel Toomey
 
PDF
A Late Bloomer's Guide to GenAI: Ethics, Bias, and Effective Prompting - Boha...
Mindy Bohannon
 
PDF
Enhancing emotion recognition model for a student engagement use case through...
IAESIJAI
 
PDF
Consumable AI The What, Why & How for Small Teams.pdf
Vivek Sai
 
PDF
From MVP to Full-Scale Product A Startup’s Software Journey.pdf
KodekX
 
PDF
Getting started with AI Agents and Multi-Agent Systems
Maxim Salnikov
 
PPTX
AI IN MARKETING- PRESENTED BY ANWAR KABIR 1st June 2025.pptx
HiraJay1
 
PDF
Five Habits of High-Impact Board Members
OnBoard
 
PDF
STKI Israel Market Study 2025 version august
Dr. Jimmy Schwarzkopf
 
PDF
TrustArc Webinar - Click, Consent, Trust: Winning the Privacy Game
TrustArc
 
PDF
Architecture types and enterprise applications.pdf
ChristopherTHyatt
 
PDF
Hybrid horned lizard optimization algorithm-aquila optimizer for DC motor
IAESIJAI
 
PPTX
Final SEM Unit 1 for mit wpu at pune .pptx
anishtilekar08
 
PDF
Zenith AI: Advanced Artificial Intelligence
Biz Preneurs
 
PPT
Geologic Time for studying geology for geologist
ssuser738f5a
 
PPTX
The various Industrial Revolutions .pptx
bandileshozi3
 
PDF
A proposed approach for plagiarism detection in Myanmar Unicode text
IAESIJAI
 
PPTX
Custom Battery Pack Design Considerations for Performance and Safety
Epec Engineered Technologies
 
PDF
Developing a website for English-speaking practice to English as a foreign la...
IAESIJAI
 
Taming the Chaos: How to Turn Unstructured Data into Decisions
Safe Software
 
Modernising the Digital Integration Hub
Daniel Toomey
 
A Late Bloomer's Guide to GenAI: Ethics, Bias, and Effective Prompting - Boha...
Mindy Bohannon
 
Enhancing emotion recognition model for a student engagement use case through...
IAESIJAI
 
Consumable AI The What, Why & How for Small Teams.pdf
Vivek Sai
 
From MVP to Full-Scale Product A Startup’s Software Journey.pdf
KodekX
 
Getting started with AI Agents and Multi-Agent Systems
Maxim Salnikov
 
AI IN MARKETING- PRESENTED BY ANWAR KABIR 1st June 2025.pptx
HiraJay1
 
Five Habits of High-Impact Board Members
OnBoard
 
STKI Israel Market Study 2025 version august
Dr. Jimmy Schwarzkopf
 
TrustArc Webinar - Click, Consent, Trust: Winning the Privacy Game
TrustArc
 
Architecture types and enterprise applications.pdf
ChristopherTHyatt
 
Hybrid horned lizard optimization algorithm-aquila optimizer for DC motor
IAESIJAI
 
Final SEM Unit 1 for mit wpu at pune .pptx
anishtilekar08
 
Zenith AI: Advanced Artificial Intelligence
Biz Preneurs
 
Geologic Time for studying geology for geologist
ssuser738f5a
 
The various Industrial Revolutions .pptx
bandileshozi3
 
A proposed approach for plagiarism detection in Myanmar Unicode text
IAESIJAI
 
Custom Battery Pack Design Considerations for Performance and Safety
Epec Engineered Technologies
 
Developing a website for English-speaking practice to English as a foreign la...
IAESIJAI
 

Writing A Foreign Data Wrapper

  • 1. Writing A Foreign Data Wrapper Bernd Helmle, bernd.helmle@credativ.de 30. Oktober 2013
  • 2. Why FDWs? ...it is in the SQL Standard (SQL/MED) ...migration ...heterogeneous infrastructure ...integration of remote (non-)relational datasources ...it’s cool http: //rhaas.blogspot.com/2011/01/why-sqlmed-is-cool.html
  • 3. Access remote datasources as PostgreSQL tables... CREATE EXTENSION IF NOT EXISTS informix_fdw; CREATE SERVER sles11_tcp FOREIGN DATA WRAPPER informix_fdw OPTIONS ( informixdir ’/Applications/IBM/informix’, informixserver ’ol_informix1170’ ); CREATE USER MAPPING FOR bernd SERVER sles11_tcp OPTIONS ( password ’informix’, username ’informix’ ); CREATE FOREIGN TABLE bar ( id integer, value text ) SERVER sles11_tcp OPTIONS ( client_locale ’en_US.utf8’, database ’test’, db_locale ’en_US.819’, query ’SELECT * FROM bar’ ); SELECT * FROM bar;
  • 4. What we need... ...a C-interface to our remote datasource ...knowledge about PostgreSQL’s FDW API ...an idea how we deal with errors ...how remote data can be mapped to PostgreSQL datatypes ...time and steadiness Python-Gurus also could use http://multicorn.org/.
  • 5. Before you start your own... Have a look at http://wiki.postgresql.org/wiki/Foreign_data_wrappers
  • 6. Let’s start... extern Datum ifx_fdw_handler(PG_FUNCTION_ARGS); extern Datum ifx_fdw_validator(PG_FUNCTION_ARGS); CREATE FUNCTION ifx_fdw_handler() RETURNS fdw_handler AS ’MODULE_PATHNAME’ LANGUAGE C STRICT; CREATE FUNCTION ifx_fdw_validator(text[], oid) RETURNS void AS ’MODULE_PATHNAME’ LANGUAGE C STRICT; CREATE FOREIGN DATA WRAPPER informix_fdw HANDLER ifx_fdw_handler VALIDATOR ifx_fdw_validator;
  • 7. FDW handler Creates and initializes a FdwRoutine structure, example: Datum ifx_fdw_handler(PG_FUNCTION_ARGS) { FdwRoutine *fdwRoutine = makeNode(FdwRoutine); fdwRoutine->ExplainForeignScan = ifxExplainForeignScan; fdwRoutine->BeginForeignScan = ifxBeginForeignScan; fdwRoutine->IterateForeignScan = ifxIterateForeignScan; fdwRoutine->EndForeignScan = ifxEndForeignScan; fdwRoutine->ReScanForeignScan = ifxReScanForeignScan; #if PG_VERSION_NUM < 90200 fdwRoutine->PlanForeignScan = ifxPlanForeignScan; #else fdwRoutine->GetForeignRelSize = ifxGetForeignRelSize; fdwRoutine->GetForeignPaths = ifxGetForeignPaths; fdwRoutine->GetForeignPlan = ifxGetForeignPlan; #endif
  • 8. FDW handler ...and much more callbacks for DML with PostgreSQL 9.3 Datum ifx_fdw_handler(PG_FUNCTION_ARGS) { [...] #if PG_VERSION_NUM >= 90300 fdwRoutine->AddForeignUpdateTargets fdwRoutine->PlanForeignModify fdwRoutine->BeginForeignModify fdwRoutine->ExecForeignInsert fdwRoutine->ExecForeignDelete fdwRoutine->ExecForeignUpdate fdwRoutine->EndForeignModify fdwRoutine->IsForeignRelUpdatable #endif PG_RETURN_POINTER(fdwRoutine); } = = = = = = = = ifxAddForeignUpdateTargets; ifxPlanForeignModify; ifxBeginForeignModify; ifxExecForeignInsert; ifxExecForeignDelete; ifxExecForeignUpdate; ifxEndForeignModify; ifxIsForeignRelUpdatable;
  • 9. FDW validator callback Called via CREATE FOREIGN TABLE or ALTER FOREIGN TABLE Validates a List * of FDW options. Use untransformRelOptions() to get a list of FDW options Don’t forget to test for duplicated options! Up to you which options you want to support Have a look into foreign/foreign.h for various helper functions
  • 10. FDW API callback routines (1) #ifdef PG_VERSION_NUM < 90200 static FdwPlan *PlanForeignScan(Oid foreignTableOid, PlannerInfo *planInfo, RelOptInfo *baserel); #else static void GetForeignRelSize(PlannerInfo *root, RelOptInfo *baserel, Oid foreignTableId); static void GetForeignPaths(PlannerInfo *root, RelOptInfo *baserel, Oid foreignTableId); static ForeignScan *GetForeignPlan(PlannerInfo *root, RelOptInfo *baserel, Oid foreignTableId, ForeignPath *best_path, List *tlist, List *scan_clauses); #endif
  • 11. FDW API callback routines (2) static void ExplainForeignScan(ForeignScanState *node, ExplainState *es); static void BeginForeignScan(ForeignScanState *node, int eflags); static TupleTableSlot *IterateForeignScan(ForeignScanState *node); static void EndForeignScan(ForeignScanState *node);
  • 12. FDW API callback routines (3) 9.2 has callbacks for ANALYZE, too: bool AnalyzeForeignTable (Relation relation, AcquireSampleRowsFunc *func, BlockNumber *totalpages); int AcquireSampleRowsFunc (Relation relation, int elevel, HeapTuple *rows, int targrows, double *totalrows, double *totaldeadrows);
  • 13. FDW API callback routines (4) 9.3 introduces callbacks for DML actions #if PG_VERSION_NUM >= 90300 fdwRoutine->AddForeignUpdateTargets fdwRoutine->PlanForeignModify fdwRoutine->BeginForeignModify fdwRoutine->ExecForeignInsert fdwRoutine->ExecForeignDelete fdwRoutine->ExecForeignUpdate fdwRoutine->EndForeignModify fdwRoutine->IsForeignRelUpdatable fdwRoutine->ExplainForeignModify #endif = = = = = = = = = ifxAddForeignUpdateTargets; ifxPlanForeignModify; ifxBeginForeignModify; ifxExecForeignInsert; ifxExecForeignDelete; ifxExecForeignUpdate; ifxEndForeignModify; ifxIsForeignRelUpdatable; ifxExplainForeignModify;
  • 14. FDW Flow AddForeignTargets() GetForeignRelSize() GetForeignPaths() Query Cost Estimates Query Planning Query Planning - Finalize Prepare / Metadata GetForeignPlan() PlanForeignModify() IsForeignRelUpdatable() BeginForeignScan() BeginForeignModify() Materialize IterateForeignScan() ExecForeignUpdate() ExecForeignDelete() ExecForeignInsert() EndForeignScan() Free resources
  • 15. AddForeignTargets static void ifxAddForeignUpdateTargets(Query *parsetree, RangeTblEntry *target_rte, Relation target_relation); Injects a custom column into the parsetree for DELETE and UPDATE commands. Can be NULL in case DELETE is not supported or UPDATE actions rely on unchanged unique keys.
  • 16. FDW Query Planning Setup and Planning a scan or modify action on a foreign datasource E.g. establish and cache remote connection Initialize required supporting structures for remote access Planner info and cost estimates via baserel and root parameters. Big differences between 9.1 and 9.2 API, DML introduced with 9.3 API
  • 17. GetForeignRelSize() (1) Size estimates for remote datasource (table size, number of rows, ...) root: Query Information Structure baserel: Table Information Structure, carry your FDW private information in baserel->fdw private. baserel->fdw private is a pointer to your state data structure.
  • 18. GetForeignRelSize() (2) baserel->rows = ifxGetEstimatedNRows(&coninfo); baserel->width = ifxGetEstimatedRowSize(&coninfo); baserel->fdw_private = (void *) planState;
  • 19. GetForeignPaths() (1) Create access path for foreign datasource. ForeignPath access path required at least. Multiple paths possible (e.g. presorted results, ...) Arbitrarily complex
  • 20. GetForeignPaths() (2) planState = (IfxFdwPlanState *) baserel->fdw_private; /* * Create a generic foreign path for now. We need to consider any * restriction quals later, to get a smarter path generation here. * * For example, it is quite interesting to consider any index scans * or sorted output on the remote side and reflect it in the * choosen paths (helps nested loops et al.). */ add_path(baserel, (Path *) create_foreignscan_path(root, baserel, baserel->rows, planState->coninfo->planData.costs, planState->coninfo->planData.costs, NIL, NULL, NIL));
  • 21. GetForeignPlan() (1) Creates a final ForeignScan plan node based on paths created by GetForeignPaths() Additional parameters ForeignPath *best path: Choosen foreign access path (best) List *tlist: Target list List *scan clauses: Restriction clauses enforced by the plan
  • 22. GetForeignPlan() (2) ForeignScan plan node should be created by make foreignscan(): ForeignScan * make_foreignscan(List *qptlist, List *qpqual, Index scanrelid, List *fdw_exprs, List *fdw_private) fdw exprs: Expressions to be evaluated by the planner fdw private: Private FDW data
  • 23. Passing FDW planning info to execution state Save parameters in the foreignScan->fdw_private pointer. Must be copieable with copyObject. Use a List * with either bytea or/and constant values (via makeConst). List *plan_values; plan_values = NIL; plan_values = lappend(plan_values, makeConst(BYTEAOID, -1, InvalidOid, -1, PointerGetDatum(ifxFdwPlanDataAsBytea(coninfo)) false, false));
  • 24. Predicate Pushdown Challenge: Filter the data on the remote dataset before transferring them Nobody wants to filter thousands of rows to just get one Idea: push filter conditions down to the foreign datasource (if possible) Ideally done during planning phase (GetForeignRelSize(), GetForeignPaths()) baserel->baserestrictinfo Hard to get it right
  • 25. Predicate Pushdown baserel->baserestrictinfo: List of predicates belonging to the foreign table (logically AND’ed) baserel->reltargetlist: List of columns belonging to the foreign table Have a look at expression tree walker() and ruleutils API (include/nodes/nodeFuncs.h, include/utils/ruleutils.h) ListCell *cell; foreach(cell, baserel->baserestrictinfo) { RestrictInfo *info; info = (RestrictInfo *) lfirst(cell); if (IsA(info->clause, OpExpr)) { /* examine right and left operand */ } }
  • 26. Predicate Pushdown, Example SELECT COUNT(*) FROM sles11.inttest; count ------10001 (1 row) EXPLAIN SELECT * FROM foo JOIN centosifx.inttest t ON (t.f1 = foo.id) WHERE t.f1 = 104 AND t.f2 = 120; QUERY PLAN --------------------------------------------------------------------------------Nested Loop (cost=2.00..1695.03 rows=1 width=18) -> Seq Scan on foo (cost=0.00..1693.01 rows=1 width=4) Filter: (id = 104) -> Foreign Scan on inttest t (cost=2.00..2.01 rows=1 width=14) Filter: ((f1 = 104) AND (f2 = 120)) Informix costs: 2.00 Informix query: SELECT * FROM inttest WHERE (f1 = 104) AND (f2 = 120) (7 rows)
  • 27. PlanForeignModify Create modify state information, e.g. get a connection, describe remote metadata, ... Private state information attached to ModifyTable plan node State information pushed down to BeginForeignModify Same with planning a scan: private state information must be in a List * format. INSERT probably needs more work here (connection preparing, ...)
  • 28. PlanForeignModify - Access scan state Possible to access the scan state created during planning phase static List * ifxPlanForeignModify(PlannerInfo *root, ModifyTable *plan, Index resultRelation, int subplan_index) { if ((resultRelation < root->simple_rel_array_size) && (root->simple_rel_array[resultRelation] != NULL)) { RelOptInfo *relInfo = root->simple_rel_array[resultRelation]; IfxFdwExecutionState *scan_state; /* * Extract the state of the foreign scan. */ scan_state = (IfxFdwExecutionState *) ((IfxFdwPlanState *)relInfo->fdw_private)->state; } }
  • 29. BeginForeignScan() void BeginForeignScan (ForeignScanState *node, int eflags); Execute startup callback for the FDW. Basically prepares the FDW for executing a scan. ForeignScanState saves function state values. Use node->fdw_state to assign your own FDW state structure. Must handle EXPLAIN and EXPLAIN ANALYZE by checking eflags & EXEC_FLAG_EXPLAIN_ONLY
  • 30. BeginForeignModify Like BeginForeignScan(), starting callback for DML actions into the executor. Depending on the DML action, might be differences in preparing stuff E.g. setup connection or get connection from cache Deserialize plan data pushed down from PlanForeignModify()
  • 31. IterateForeignScan() (1) TupleTableSlot * IterateForeignScan (ForeignScanState *node); Fetches data from the remote source. Data conversion Materializes a physical or virtual tuple to be returned. Needs to return an empty tuple when done. Private FDW data located in node->fdw state
  • 32. IterateForeignScan() (2) Returning a virtual tuple TupleTableSlot *slot = node->ss.ss_ScanTupleSlot; slot->tts_isempty = false; slot->tts_nvalid = number_cols;; slot->tts_values = (Datum *)palloc(sizeof(Datum) * slot->tts_nvalid); slot->tts_isnull = (bool *)palloc(sizeof(bool) * slot->tts_nvalid); for (i = 0; j < attrCount - 1; i) { tupleSlot->tts_isnull[i] = false; tupleSlot->tts_values[i] = PointerGetDatum(val); }
  • 33. ReScanForeignScan() void ReScanForeignScan (ForeignScanState *node); Prepares the FDW to handle a rescan Begins the scan from the beginning, e.g when used in scrollable cursors Must take care for changed query parameters! Better to just “instruct” IterateForeignScan() to do the right thing (tm)
  • 34. Execute a modify action(1) static TupleTableSlot * ifxExecForeignUpdate(EState *estate, ResultRelInfo *rinfo, TupleTableSlot *slot, TupleTableSlot *planSlot) static TupleTableSlot * ifxExecForeignInsert(EState *estate, ResultRelInfo *rinfo, TupleTableSlot *slot, TupleTableSlot *planSlot); static TupleTableSlot * ifxExecForeignDelete(EState *estate, ResultRelInfo *rinfo, TupleTableSlot *slot, TupleTableSlot *planSlot);
  • 35. Execute a modify action (2) ExecForeignUpdate, ExecForeignDelete and ExecForeignInsert callbacks Called for every tuple to work on An INSERT action doesn’t employ a remote scan! Does all the necessary data conversion if necessary slot carries all information for the target tuple Private FDW data stored in rinfo->ri FdwState
  • 36. EndForeignScan(), EndForeignModify() void EndForeignScan (ForeignScanState *node); void EndForeignModify(EState *estate, ResultRelInfo *rinfo) EndForeignModify() run when no more tuple to modify EndForeignScan() run when IterateForeignScan returns no more rows, but after EndForeignModify() Finalizes the remote scan and modify action Close result sets, handles, connection, free memory, etc...
  • 37. Transaction Callbacks Connect local transactions with remote transactions and savepoints RegisterXactCallback(pgfdw_xact_callback, NULL); RegisterSubXactCallback(pgfdw_subxact_callback, NULL); static void fdw_xact_callback(XactEvent event, void *arg); static void fdw_subxact_callback(SubXactEvent event, SubTransactionId mySubid, SubTransactionId parentSubid, void *arg)
  • 38. Memory Management PostgreSQL uses palloc() Memory is allocated in CurrentMemoryContext Use your own MemoryContext where necessary (e.g. IterateForeignScan()) Memory allocated in external libraries need special care
  • 39. Data conversion Easy, if the remote datasource delivers a well formatted value string (e.g. date strings formatted as yyyy-mm-dd). Use type input function directly Binary compatible types (e.g integer) Binary data should always be bytea String data must have a valid encoding!
  • 40. Data conversion - Encoding Within a FDW, a backend acts like any other client: ensure encoding compatibility or encode your string data properly. A look at mb/pg wchar.h might be of interest. GetDatabaseEncoding() pg_do_encoding_conversion()
  • 41. Data conversion - Get type input function regproc typinputfunc; Datum result; HeapTuple type_tuple; type_tuple = SearchSysCache1(TYPEOID, inputOid); if (!HeapTupleIsValid(type_tuple)) { /* * Oops, this is not expected... */ ... } ReleaseSysCache(type_tuple); typinputfunc = ((Form_pg_type) GETSTRUCT(type_tuple))->typinput; result = OidFunctionCall2(typinputfunc, CStringGetDatum(buf), ObjectIdGetDatum(InvalidOid));
  • 42. Error Handling (1) Set FDW SQLSTATE according to your error condition class HV, see http: // www. postgresql. org/ docs/ 9. 3/ static/ errcodes-appendix. html Alternative: map remote error conditions to PostgreSQL errors Be careful with elog(ERROR, ...).
  • 43. Error Handling (2) Example (there is no FDW WARNING SQLSTATE): if (err == IFX_CONNECTION_WARN) { IfxSqlStateMessage message; ifxGetSqlStateMessage(1, &message); ereport(WARNING, (errcode(WARNING), errmsg("opened informix connection with warnings"), errdetail("informix SQLSTATE %s: "%s"", message.sqlstate, message.text))); }
  • 44. Catching Errors (1) Sometimes necessary to catch backend errors Synchronize error conditions between PostgreSQL and remote datasource Possibility: use a PG TRY...PG CATCH block.
  • 45. Catching Errors (2) PG_TRY(); { ... typinputfunc = getTypeInputFunction(state, PG_ATTRTYPE_P(state, attnum)); result = OidFunctionCall2(typinputfunc, CStringGetDatum(pstrdup(buf)), ObjectIdGetDatum(InvalidOid)); } PG_CATCH(); { ifxRewindCallstack(&(state->stmt_info)); PG_RE_THROW(); } PG_END_TRY();