![politicians, who, however good their intentions, proved that they had
not yet grasped the difference between a quarrel at law and a quarrel
at war.
It left the nation disappointed. The people felt we had been fooling
with the war too long; that the time had arrived for some strong and
decisive action. That politics and patronage should be shelved and
the Navy given a free hand. It remembered how the Government had
hesitated, procrastinated, and vacillated in this so-called blockade, as
in other matters. It remembered that Parliament had refused to pass
a code of international rules called the Declaration of London because
that code, made largely to please Germany, weakened the hands of
the Navy. It remembered that the Government had gone behind the
back of Parliament and illegally put that very code into operation
after war began. It had not forgotten that this proved such a
scandalous weakening of our right and our strength that soon after
the Coalition Government came into being that code was said to have
been scrapped. Even as to this doubts arose for long afterwards.[21]
It had not forgotten the seventeen long months of public pressure
and the trouble there had been to force cotton as contraband; nor
the seventeen months of "wait and see" before the Navy was
permitted to examine mails and extract (inter alia) parcels of rubber.
It had not forgotten Sir Edward Grey's declaration that "he had no
intention of making cotton contraband"; nor Lord Haldane's
contention that "it was useless stopping the import of cotton to
Germany, because if we did Germany could find a substitute for it."
The nation had been deceived and lulled to sleep before by soft
words and gentle assurances. It had been told, "we decline to be
bound by judicial niceties." It had been promised "to prevent
commodities of any kind from entering or leaving the enemy's
country"; "to stick at nothing." It remembered with some misgiving
how these promises had been kept.[22]
What, it reasoned, were the disappointments of a few Dutch and
Scandinavian adventurers from making fortunes out of a war which to](https://image.slidesharecdn.com/1234761-250513034627-4678efc1/85/Pentaho-Data-Integration-4-Cookbook-Pulvirenti-Adrin-Sergio-57-320.jpg)
![authorities) I must not collect these figures, although to me and
others working with me they were comparatively easy of access.
So far as Foreign Office knowledge is concerned, it is hardly a credit
to the ability or even sanity of the British Legations in Scandinavia if
they have denied knowledge of these colossal imports of goods into
Germany, which were known to almost every inhabitant of seaport
towns. If they deliberately shut their eyes to the evidence all around
them, they presumably obeyed orders. One could then only wonder
as to the reason for such suicidal policy.
As before mentioned, at the commencement of his speech Sir Edward
Grey laid stress upon the fact that part of the stated increased
import, namely, 2,000,000 barrels of flour were allowed to be
exported to Belgium; whilst a little later in his speech he admitted
that "She [Germany] had requisitioned the food supplies of the civil
population of Poland and Belgium." Almost immediately afterwards
Lord Robert Cecil strove hard to back up the Secretary for Foreign
Affairs, but he could not give the House any positive assurance that
the Belgian Relief distribution was absolutely independent of German
control. The disposition of this is therefore obvious.
Sir Edward Grey attempted to whittle down the U.S.A. exports of
wheat by stating that nearly half went to Spain, Portugal, Greece,
and Malta; but he did not refer to the corn, etc., exported to Northern
neutrals from Liverpool and other British ports, nor did he make any
allowances for the stream of mysterious ships sailing round far
northern seas (many of them choosing the passage north of Iceland),
which sighted land on the north-western coast of Norway and carried
their course inside neutral waters into the Baltic; which heavily-laden
cargo-boats I and others in the Secret Service had watched and
reported week by week and month by month with heart-rending
persistency. The majority of these ships probably sailed direct to
German ports, and no records of their cargoes were likely to be
made, or returned from any country concerning them. Nor did Sir
Edward Grey make reference to the grain ships, which although
nominally bound for Scandinavian ports, were intercepted by their](https://image.slidesharecdn.com/1234761-250513034627-4678efc1/85/Pentaho-Data-Integration-4-Cookbook-Pulvirenti-Adrin-Sergio-59-320.jpg)
![caretakers, and even charwomen and servant-girls, our own inflated
home prices of necessities and commodities—Sir Edward Grey's
answer to all this was: The Government had lately sent Lord
Faringdon to examine the position in Holland and Scandinavia and he
reported that on the whole things were very satisfactory and that all
was being done that could be done to prevent the enemy obtaining
supplies.
Well might the fat stomachs of the "Goulashes"[23] extend and shake
in merriment when they read these comfortable words!
Sir Edward Grey concluded his speech with this stirring peroration:
The whole of our resources were engaged in this war, and our
maximum effort was at the disposal of our Allies in carrying on this
conflict. With them we should see it through to the end and we
should slacken no effort in the common cause. We should exert all
our efforts to put the maximum possible pressure upon the enemy,
and part of that pressure must be doing the most we could to
prevent supplies going to or from the enemy, using the Navy to its
full power ... and in common with our Allies sparing nothing, whether
it were military, naval, or financial effort, which this country could
afford, to see the thing through with them to the end.
In the loud cheering with which the House of Commons received the
speech no thought was given to the famous words of Napoleon: "Put
no faith in talk which is not borne out by action"; whilst future events
went to show that Napoleon truly forecasted England's present-day
weakness when he wrote: "Feebleness in its Government is the most
frightful calamity that can befall a nation."
Contrast Sir Edward Grey's eloquent words and diplomatic
evasiveness upon the treatment of neutrals with the plain,
outspoken, thoroughly English opinion of Lord Fisher, who is credited
with having said:
"There are no such things as neutral powers. Powers are either with
us or against us. If they are friendly they will put up with some](https://image.slidesharecdn.com/1234761-250513034627-4678efc1/85/Pentaho-Data-Integration-4-Cookbook-Pulvirenti-Adrin-Sergio-63-320.jpg)
![inconvenience; if they are unfriendly they will squeal. Let them
squeal."
Had we acted throughout on this dictum the war would most
probably have been over well inside of eighteen months. Men of the
calibre of this grand old Sea Lord, whose farsight, foresight, and
second sight have endeared him to the nation and made him unique
and incomparable, would soon have made short work of the war. Yet
they were not wanted by the then present-day party-system
Government. They were much too blunt and honest and energetically
active.
The nation will also remember that when Lord Kitchener of Khartoum
returned from the East in the early days of the then present
Government, it had no use for his invaluable services. He was actually
permitted to accept a directorship of one of our poorest railway
companies on the south coast for want of a better occupation.[24]
But the Press and the public soon brought the Government to book,
as it seemingly had to do in every matter of real national importance.
The Government tried to keep Lord Haldane installed at the War
Office, but the Press would have none of it. It also insisted on K. of K.
being placed in his proper place and kept there. More's the pity that
he was not given a free hand to do as he liked.
The Press also clamoured for Lord Fisher as First Lord of the
Admiralty. The nation knows how he was treated. A captain in the
Navy aptly described the unwanted and slighted Admiral expert in
John Bull, February, 1916, as follows:
"Lord John Fisher is to-day our second Nelson—a diplomatist
among diplomats and a strategist unequalled in our history. What
has Lord John Fisher done?
"He scrapped 162 obsolete warships which were rotting in
harbour at great expense—for which the Government tried to
reprimand him.](https://image.slidesharecdn.com/1234761-250513034627-4678efc1/85/Pentaho-Data-Integration-4-Cookbook-Pulvirenti-Adrin-Sergio-64-320.jpg)
![Pig Iron
Tons
1915 5,530,000
1916 6,497,000
Steel
1915 6,187,000
1916 7,756,000
The Lokal Anzeiger, July 28th, 1916, remarked: "These figures
constitute a most gratifying state of affairs in respect of the
requirements of the German Armies." No wonder the captured
German officer remarked: "You English will always be fools, whilst we
Germans can never be gentlemen"!
In August[25] a Mr. E. Bell, of 12, Yarborough Road, Lincoln, wrote to
the Press as follows:
"The talk of tightening the blockade of Germany is rather futile in
face of the following Board of Trade figures referring to cotton
yarn exported from the United Kingdom to the following neutral
countries:
June Sweden Norway Denmark Holland Switzerland
1914 108,900 218,700 106,400 3,220,800 722,600
1915 260,800 348,300 204,700 4,493,300 1,788,800
1916 279,200 508,200 598,400 7,539,800 1,304,100
"Germany is obviously getting the surplus."
The values[26] of New York exports taken for the week July 30th to
August 5th are equally startling:
1915. 1916.
New York to £ £
Norway 1,884 137,176](https://image.slidesharecdn.com/1234761-250513034627-4678efc1/85/Pentaho-Data-Integration-4-Cookbook-Pulvirenti-Adrin-Sergio-83-320.jpg)
![Holland 713 717,601
Holland and Scandinavia 123,327 970,255
On August 26th, 1916, an agreement was signed between the Dutch
Fishing Association and the British Government regarding the release
of some 120 to 150 Dutch fishing-boats laid up in Scottish ports,
whereby not more than 20 per cent. of their catch shall be permitted
to go to Germany. Of the remainder twenty per cent. was to be
retained for home consumption, and sixty per cent. sold to neutral
countries. On each barrel of this sixty per cent. the good, kind,
benevolent British Government agreed to pay a subsidy of 30s. to the
Dutch boat-owners.
Now the D.F.A. owned about 850 vessels and 1,000 barrels is a good
average season's catch!
In addition to this arrangement the British Government agreed to pay
full compensation for their loss of part of the season, to be calculated
on the basis of the returns on an average season. They also agreed
to pay for any damage which might have happened to the interned
boats.[27]
One wonders what British fishermen whose vessels have been
commandeered had to say when they were informed of these facts.
The Hamburger Nachrichten of August 23rd, 1916, published a
telegram from its Hague correspondent declaring that the semi-
official German Central Purchase Company was seizing Dutch food in
enormous quantities; that local merchants were in a state of alarm
and threatening Government interference; and their correspondent
defiantly stated: "The Netherlands Government will hardly dream of
interfering with the activity of the Dutch Bureau of the German
Central Purchase Company, the operations of which are assuming
larger and larger dimensions."
To add further proof of the utter futility and hollow sham of the
alleged blockade safeguards, namely, the Danish Association
Agreement and the Netherlands Overseas Trust, Sir Henry Dalziel](https://image.slidesharecdn.com/1234761-250513034627-4678efc1/85/Pentaho-Data-Integration-4-Cookbook-Pulvirenti-Adrin-Sergio-84-320.jpg)
![Of the late Prime Minister (Mr. Asquith) one able editor wrote:
"Never before in all our history have such opportunities been
given. He had no opposition; the nation was solid; the Empire
was behind him. No country has ever given any leader such
devotion and none has ever seen its devotion so carelessly
wasted. Declaring he would 'stick at nothing,' he stuck at
everything, and moved only when he was pushed."[28]
What Germany thought of the change is reflected in an extract from
its Press when it first heard of the resignation of Mr. David Lloyd
George from the War Office, and it was under the belief that the
Haldane group had triumphed over him.
The Bavarian Courier, December 5th, 1916, said: "This is a terrible
disaster for the war party in England," whilst the Leipzig Tageblatt
said: "The British people have doubtless had enough of this war
agitator. His fall from power brings nearer an honourable peace for
Germany."
Within a few days of Mr. Lloyd George being created Prime Minister of
England the Kaiser was seeking peace. Res ipsa loquitur.
* * * * * *
What has been given is merely a rough and very deficient resumé of
England's sham blockade, which was permitted to muddle along its
costly, tragic, and fatal course until the Americans joined the Allies in
their fight for freedom and the rights of small nations. Washington at
once swept aside maudlin sentiment by its practical common sense,
get-right-there-quick decisions.
The nation's relief cannot be expressed in words.
Was it to be wondered at that from the soul of the Motherland
prayers had so long and so often ascended?
"Oh, for a man of the old, old Viking blood to lead and direct the
battle in place of those poor craven lawyer politicians in the Cabinet](https://image.slidesharecdn.com/1234761-250513034627-4678efc1/85/Pentaho-Data-Integration-4-Cookbook-Pulvirenti-Adrin-Sergio-89-320.jpg)
Pentaho Data Integration 4 Cookbook Pulvirenti Adrin Sergio
- 1. Pentaho Data Integration 4 Cookbook Pulvirenti Adrin Sergio download https://ebookbell.com/product/pentaho-data- integration-4-cookbook-pulvirenti-adrin-sergio-2469522 Explore and download more ebooks at ebookbell.com
- 2. Here are some recommended products that we believe you will be interested in. You can click the link to download. Pentaho Data Integration Cookbook Second Edition 2nd New Edition Alex Meadows https://ebookbell.com/product/pentaho-data-integration-cookbook- second-edition-2nd-new-edition-alex-meadows-4656442 Pentaho Data Integration Beginners Guide 2nd Mara Carina Roldn https://ebookbell.com/product/pentaho-data-integration-beginners- guide-2nd-mara-carina-roldn-6809440 Learning Pentaho Data Integration 8 Ce 3rd Edition Mara Carina Roldn https://ebookbell.com/product/learning-pentaho-data- integration-8-ce-3rd-edition-mara-carina-roldn-7410824 Pentaho Kettle Solutions Building Open Source Etl Solutions With Pentaho Data Integration 1st Edition Matt Casters https://ebookbell.com/product/pentaho-kettle-solutions-building-open- source-etl-solutions-with-pentaho-data-integration-1st-edition-matt- casters-2265962
- 3. Pentaho 32 Data Integration Beginners Guide Mara Carina Roldan https://ebookbell.com/product/pentaho-32-data-integration-beginners- guide-mara-carina-roldan-2310106 Pentaho For Big Data Analytics Manoj R Patil Feris Thia https://ebookbell.com/product/pentaho-for-big-data-analytics-manoj-r- patil-feris-thia-4681270 Pentaho Solutions Business Intelligence And Data Warehousing With Pentaho And Mysql Roland Bouman Jos Van Dongen https://ebookbell.com/product/pentaho-solutions-business-intelligence- and-data-warehousing-with-pentaho-and-mysql-roland-bouman-jos-van- dongen-4105414 Pentaho Analytics For Mongodb Cookbook 1st Edition Latino Joel https://ebookbell.com/product/pentaho-analytics-for-mongodb- cookbook-1st-edition-latino-joel-55144806 Pentaho 8 Reporting For Java Developers Create Pixelperfect Analytical Reports Using Reporting Tools Francesco Corti https://ebookbell.com/product/pentaho-8-reporting-for-java-developers- create-pixelperfect-analytical-reports-using-reporting-tools- francesco-corti-55588290
- 6. Pentaho Data Integration 4 Cookbook Over 70 recipes to solve ETL problems using Pentaho Kettle Adrián Sergio Pulvirenti María Carina Roldán BIRMINGHAM - MUMBAI
- 7. Pentaho Data Integration 4 Cookbook Copyright © 2011 Packt Publishing All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews. Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the authors, nor Packt Publishing, and its dealers and distributors will be held liable for any damages caused or alleged to be caused directly or indirectly by this book. Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information. First published: June 2011 Production Reference: 1170611 Published by Packt Publishing Ltd. 32 Lincoln Road Olton Birmingham, B27 6PA, UK. ISBN 978-1-849515-24-5 www.packtpub.com Cover Image by Ed Maclean (edmaclean@gmail.com)
- 8. Credits Authors Adrián Sergio Pulvirenti María Carina Roldán Reviewers Jan Aertsen Pedro Alves Slawomir Chodnicki Paula Clemente Samatar Hassan Nelson Sousa Acquisition Editor Usha Iyer Development Editor Neha Mallik Technical Editors Conrad Sardinha Azharuddin Sheikh Project Coordinator Joel Goveya Proofreaders Stephen Silk Aaron Nash Indexer Tejal Daruwale Graphics Nilesh Mohite Production Coordinator Kruthika Bangera Cover Work Kruthika Bangera
- 9. About the Authors Adrián Sergio Pulvirenti was born in Buenos Aires, Argentina, in 1972. He earned his Bachelor's degree in Computer Sciences at UBA, one of the most prestigious universities in South America. He has dedicated more than 15 years to developing desktop and web-based software solutions. Over the last few years he has been leading integration projects and development of BI solutions. I'd like to thank my lovely kids Camila and Nicolas, who understood that I couldn't share with them the usual videogame sessions during the writing process. I'd also thank my wife who introduced me to the Pentaho world. María Carina Roldán was born in Esquel, Argentina, in 1970. She earned her Bachelors degree in Computer Science at UNLP in La Plata; after that she did a postgraduate course in Statistics at the University of Buenos Aires (UBA) in Buenos Aires city where she lives since 1994. She has worked as a BI consultant for more than 10 years. Over the last four years, she has been dedicated full time to developing BI solutions using Pentaho Suite. Currently she works for Webdetails, one of the main Pentaho contributors. She is the author of Pentaho 3.2 Data Integration: Beginner's Guide published by Packt Publishing in April 2010. You can follow her on Twitter at @mariacroldan.
- 10. I'd like to thank those who have encouraged me to write this book: On one hand, the Pentaho community. They have given me a rewarding feedback after the Beginner's book. On the other side, my husband who without hesitation agreed to write the book with me. Without them I'm not sure I would have embarked on a new book project. I'd also like to thank the technical reviewers for the time and dedication that they have put in reviewing the book. In particular, thanks to my colleagues at Webdetails; it's a pleasure and a privilege to work with them every day.
- 11. About the Reviewers Jan Aertsen has worked in IT and decision support for the past 10 years. Since the beginning of his career he has specialized in data warehouse design and business intelligence projects. He has worked on numerous global data warehouse projects within the fashion industry, retail, banking and insurance, telco and utilities, logistics, automotive, and public sector. Jan holds the degree of Commercial Engineer in international business affairs from the Catholic University of Leuven (Belgium) and extended his further knowledge in the field of business intelligence through a Masters in Artificial Intelligence. In 1999 Jan started up the business intelligence activities at IOcore together with some of his colleagues, rapidly making this the most important revenue area of the Belgian affiliate. They quickly gained access to a range of customers as KPN Belgium, Orange (now Base), Mobistar, and other Belgian Telcos. After this experience Jan joined Cap Gemini Ernst & Young in Italy and rapidly became one of their top BI project managers. After having managed some large BI projects (up to 1 million € projects) Jan decided to leave the company and pursue his own ambitions. In 2002, he founded kJube as an independent platform to develop his ambitions in the world of business intelligence. Since then this has resulted in collaborations with numerous companies as Volvo, Fendi-LVMH, ING, MSC, Securex, SDWorx, Blinck, and Beate Uhse. Over the years Jan has worked his way through every possible aspect of business intelligence from KPI and strategy definition over budgeting, tool selection, and software investments acquisition to project management and all implementation aspects with most of the available tools. He knows the business side as well as the IT side of the business intelligence, and therefore is one of the rare persons that are able to give you a sound, all-round, vendor-independent advice on business intelligence. He continues to share his experiences in the field through his blog (blog.kjube.be) and can be contacted at jan.aertsen@kjube.be.
- 12. Pedro Alves, is the founder of Webdetails. A Physicist by formation, serious video gamer, volleyball player, open source passionate, and dad of two lovely children. Since his early professional years he has been responsible for Business Software development and his career led him to work as a Consultant in several Portuguese companies. In 2008 he decided it was time to get his accumulated experience and share his knowledge about the Pentaho Business Intelligence platform on his own. He founded Webdetails and joined the Mozilla metrics team. Now he leads an international team of BI Consultants and keeps nurturing Webdetails as a world reference Pentaho BI solutions provider and community contributor. He is the Ctools (CDF, CDA, CDE, CBF, CST, CCC) architect and, on a daily basis, keeps developing and improving new components and features to extend and maximize Pentaho's capabilities. Slawomir Chodnicki specializes in data warehousing and ETL, with a background in web development using various programming languages and frameworks. He has established his blog at http://type-exit.org to help fellow BI developers embrace the possibilities of PDI and other open source BI tools. I would like to thank all regular members of the ##pentaho IRC channel for their endless patience and support regarding PDI related questions. Very special thanks go to María Carina and Adrián Sergio for creating the Kettle Cookbook and inviting me to be part of the project. Paula Clemente was born in Sintra, Portugal, in 1983. Divided between the idea of spending her life caring about people and animals or spending quality time with computers, she started studying Computer Science at IST Engineering College—"the Portuguese MIT"—at a time where Internet Social Networking was a synonym of IRC. She graduated in 2008 after completing her Master thesis on Business Processes Management. Since then she is proudly working as a BI Consultant for Webdetails, a Portuguese company specialized in delivering Pentaho BI solutions that earned the Pentaho "Best Community Contributor 2011" award.
- 13. Samatar Hassan is an application developer focusing on data integration and business intelligence. He was involved in the Kettle project since the year it was open sourced. He tries to help the community by contributing in different ways; taking the translation effort for French language, participating in the forums, resolving bugs, and adding new features to the software. He contributed to the "Pentaho Kettle Solutions" book edited by Wiley and written by Matt Casters, the founder of Kettle. I would first like to thank Adrián Sergio and María Carina Roldán for taking the time to write this book. It is a great idea to show how to take advantage of Kettle through step-by-step recipes. Kettle users have their own ETL bible now. Finally, I'd like to thank all community members. They are the real power of open source software. Nelson Sousa is a business intelligence consultant at Webdetails. He's part of the Metrics team at Mozilla where he helps develop and maintain Mozilla's Pentaho server and solution. He specializes in Pentaho dashboards using CDF, CDE, and CDA and also in PDI, processing vast amounts of information that are integrated daily in the various dashboards and reports that are part of the Metrics team day-to-day life.
- 14. www.PacktPub.com Support files, eBooks, discount offers and more You might want to visit www.PacktPub.com for support files and downloads related to your book. Did you know that Packt offers eBook versions of every book published, with PDF and ePub files available? You can upgrade to the eBook version at www.PacktPub.com and as a print book customer, you are entitled to a discount on the eBook copy. Get in touch with us at service@packtpub.com for more details. At www.PacktPub.com, you can also read a collection of free technical articles, sign up for a range of free newsletters and receive exclusive discounts and offers on Packt books and eBooks. Why Subscribe? f Fully searchable across every book published by Packt f Copy and paste, print and bookmark content f On demand and accessible via web browser Free Access for Packt account holders If you have an account with Packt at www.PacktPub.com, you can use this to access PacktLib today and view nine entirely free books. Simply use your login credentials for immediate access.
- 16. We dedicate this book to our family and specially our adorable kids. - María Carina and Adrián -
- 18. Table of Contents 1 5 5 7 11 13 Getting data from a database by running a query built at runtime 16 Inserting or updating rows in a table 18 Inserting new rows where a simple primary key has to be generated 23 Inserting new rows where the primary key has to be generated based on stored values 26 Deleting data from a table 30 Creating or altering a database table from PDI (design time) 34 Creating or altering a database table from PDI (runtime) 37 Inserting, deleting, or updating a table depending on a field 39 Changing the database connection at runtime 43 Loading a parent-child table 46 Chapter 2: Reading and Writing Files 51 51 52 56 57 64 66 69 71 74 77
- 19. ii 79 82 82 86 89 93 93 94 97 103 106 108 112 114 121 125 125 126 129 131 135 137 139 141 144 149 149 150 154 157 160 165 169 173 175 Introduction 175 Splitting a stream into two or more streams based on a condition 176 Merging rows of two streams with the same or different structures 184 Comparing two streams and generating differences 189
- 20. iii Table of Contents 194 198 201 204 208 Chapter 7: Executing and Reusing Jobs and Transformations 213 213 217 Executing a job or a transformation from a job by setting arguments and parameters dynamically 220 Executing a job or a transformation whose name is determined at runtime 223 Executing part of a job once for every row in a dataset 225 Executing part of a job several times until a condition is true 231 Creating a process flow 236 Moving part of a transformation to a subtransformation 242 Chapter 8: Integrating Kettle and the Pentaho Suite 249 249 Creating a Pentaho report with data coming from PDI 252 Configuring the Pentaho BI Server for running PDI jobs and transformations 257 Executing a PDI transformation as part of a Pentaho process 259 Executing a PDI job from the Pentaho User Console 266 Generating files from the PUC with PDI and the CDA plugin 270 Populating a CDF dashboard with data coming from a PDI transformation 276 Chapter 9: Getting the Most Out of Kettle 283 283 283 288 293 302 Working with Json files 305 Getting information about transformations and jobs (file-based) 308 Getting information about transformations and jobs (repository-based) 313 Appendix: Data Structures 319 319 320 321 322 323
- 22. Preface Pentaho Data Integration (PDI, also called Kettle), one of the data integration tools leaders, is broadly used for all kind of data manipulation, such as migrating data between applications or databases, exporting data from databases to flat files, data cleansing, and much more. Do you need quick solutions to the problems you face while using Kettle? Pentaho Data Integration 4 Cookbook explains Kettle features in detail through clear and practical recipes that you can quickly apply to your solutions. The recipes cover a broad range of topics including processing files, working with databases, understanding XML structures, integrating with Pentaho BI Suite, and more. Pentaho Data Integration 4 Cookbook shows you how to take advantage of all the aspects of Kettle through a set of practical recipes organized to find quick solutions to your needs. The initial chapters explain the details about working with databases, files, and XML structures. Then you will see different ways for searching data, executing and reusing jobs and transformations, and manipulating streams. Further, you will learn all the available options for integrating Kettle with other Pentaho tools. Pentaho Data Integration 4 Cookbook has plenty of recipes with easy step-by-step instructions to accomplish specific tasks. There are examples and code that are ready for adaptation to individual needs. Learn to solve data manipulation problems using the Pentaho Data Integration tool Kettle. What this book covers Chapter 1, Working with Databases helps you to deal with databases in Kettle. The recipes cover creating and sharing connections, loading tables under different scenarios, and creating dynamic SQL statements among others topics. Chapter 2, Reading and Writing Files shows you not only the basics for reading and writing files, but also all the how-tos for dealing with files. The chapter includes parsing unstructured files, reading master/detail files, generating multi-sheet Excel files, and more.
- 23. Preface 2 Chapter 3, Manipulating XML Structures teaches you how to read, write, and validate XML data. It covers both simple and complex XML structures. Chapter 4, File Management helps you to pick and configure the different options for copying, moving, and transferring lists of files or directories. Chapter 5, Looking for Data explains the different methods for searching information in databases, text files, web services, and more. Chapter 6, Understanding Data Flows focuses on the different ways for combining, splitting, or manipulating streams or flows of data in simple and complex situations. Chapter 7, Executing and Reusing Jobs and Transformations explains in a simple fashion topics that are critical for building complex PDI projects. For example, building reusable jobs and transformations, iterating the execution of a transformation over a list of data and transferring data between transformations. Chapter 8, Integrating Kettle and the Pentaho Suite. PDI aka Kettle is part of the Pentaho Business Intelligent Suite. As such, it can be used interacting with other components of the suite, for example as the datasource for reporting, or as part of a bigger process. This chapter shows you how to run Kettle jobs and transformations in that context. Chapter 9, Getting the Most Out of Kettle covers a wide variety of topics, such as customizing a log file, sending e-mails with attachments, or creating a custom functionality. Appendix, Data Structures describes some structures used in several recipes throughout the book. What you need for this book PDI is a multiplatform tool, meaning that you will be able to install the tool no matter what your operating system is. The only prerequisite to work with PDI is to have JVM 1.5 or a higher version installed. It is also useful to have Excel or Calc, a nice text editor, and access to a database engine of your preference. Having an Internet connection while reading is extremely useful as well. Several links are provided throughout the book that complement what is explained. Besides, there is the PDI forum where you may search or post doubts if you are stuck with something. Who this book is for If you are a software developer or anyone involved or interested in developing ETL solutions, or in general, doing any kind of data manipulation, this book is for you. It does not cover PDI basics, SQL basics, or database concepts. You are expected to have a basic understanding of the PDI tool, SQL language, and databases.
- 24. Preface 3 Conventions In this book, you will find a number of styles of text that distinguish between different kinds of information. Here are some examples of these styles, and an explanation of their meaning. Code words in text are shown as follows: "Copy the .jar file containing the driver to the libext/JDBC directory inside the Kettle installation directory". A block of code is set as follows: NUMBER, LASTNAME, FIRSTNAME, EXT, OFFICE, REPORTS, TITLE 1188, Firrelli, Julianne,x2174,2,1143, Sales Manager 1619, King, Tom,x103,6,1088,Sales Rep When we wish to draw your attention to a particular part of a code block, the relevant lines or items are set in bold: <request> <type>City</type> <query>Buenos aires, Argentina</query> <preferredScale>C</preferredScale> </request> New terms and important words are shown in bold. Words that you see on the screen, in menus or dialog boxes for example, appear in the text like this: "Add a Delete file entry from the File management category" Warnings or important notes appear in a box like this. Tips and tricks appear like this. Reader feedback Feedback from our readers is always welcome. Let us know what you think about this book—what you liked or may have disliked. Reader feedback is important for us to develop titles that you really get the most out of. To send us general feedback, simply send an e-mail to feedback@packtpub.com, and mention the book title via the subject of your message. If there is a book that you need and would like to see us publish, please send us a note in the SUGGEST A TITLE form on www.packtpub.com or e-mail suggest@packtpub.com.
- 25. Preface 4 If there is a topic that you have expertise in and you are interested in either writing or contributing to a book, see our author guide on www.packtpub.com/authors. Customer support Now that you are the proud owner of a Packt book, we have a number of things to help you to get the most from your purchase. Downloading the example code You can download the example code files for all Packt books you have purchased from your account at http://www.PacktPub.com. If you purchased this book elsewhere, you can visit http://www.PacktPub.com/support and register to have the files e-mailed directly to you. Errata Although we have taken every care to ensure the accuracy of our content, mistakes do happen. If you find a mistake in one of our books—maybe a mistake in the text or the code—we would be grateful if you would report this to us. By doing so, you can save other readers from frustration and help us improve subsequent versions of this book. If you find any errata, please report them by visiting http://www.packtpub.com/support, selecting your book, clicking on the errata submission form link, and entering the details of your errata. Once your errata are verified, your submission will be accepted and the errata will be uploaded on our website, or added to any list of existing errata, under the Errata section of that title. Any existing errata can be viewed by selecting your title from http://www.packtpub.com/support. Piracy Piracy of copyright material on the Internet is an ongoing problem across all media. At Packt, we take the protection of our copyright and licenses very seriously. If you come across any illegal copies of our works, in any form, on the Internet, please provide us with the location address or website name immediately so that we can pursue a remedy. Please contact us at copyright@packtpub.com with a link to the suspected pirated material. We appreciate your help in protecting our authors, and our ability to bring you valuable content. Questions You can contact us at questions@packtpub.com if you are having a problem with any aspect of the book, and we will do our best to address it.
- 26. 1 Working with Databases In this chapter, we will cover: f Connecting to a database f Getting data from a database f Getting data from a database by providing parameters f Getting data from a database by running a query built at runtime f Inserting or updating rows in a table f Inserting new rows when a simple primary key has to be generated f Inserting new rows when the primary key has to be generated based on stored values f Deleting data from a table f Creating or altering a table from PDI (design time) f Creating or altering a table from PDI (runtime) f Inserting, deleting, or updating a table depending on a field f Changing the database connection at runtime f Loading a parent-child table Introduction Databases are broadly used by organizations to store and administer transactional data such as customer service history, bank transactions, purchases and sales, and so on. They also constitute the storage method for data warehouses, the repositories used in Business Intelligence solutions.
- 27. Working with Databases 6 In this chapter, you will learn to deal with databases in Kettle. The first recipe tells you how to connect to a database, which is a prerequisite for all the other recipes. The rest of the chapter teaches you how to perform different operations and can be read in any order according to your needs. The focus of this chapter is on relational databases (RDBMS). Thus the term database is used as a synonym for relational databases through the recipes. Sample databases Through the chapter you will use a couple of sample databases. Those databases can be created and loaded by running the scripts available at the book's website. The scripts are ready to run under MySQL. If you work with a different DBMS you may have to modify the scripts slightly. For more information about the structure of the sample databases and the meaning of the tables and fields, please refer to Appendix, Data Structures. Feel free to adapt the recipes to different databases. You could try some well known databases; for example Foodmart (available as part of the Mondrian distribution at http://sourceforge.net/projects/mondrian/) or the MySQL sample databases (available at http://dev.mysql.com/doc/index-other.html). Pentaho BI platform databases As part of the sample databases used in this chapter you will use the Pentaho BI platform Demo databases. The Pentaho BI Platform Demo is a pre-configured installation that lets you explore the capabilities of the Pentaho platform. It relies on the following databases: Database name Description hibernate Administrative information including user authentication and authorization data. quartz Repository for Quartz, the scheduler used by Pentaho. sampledata Data for Steel Wheels, a fictional company that sells all kind of scale replicas of vehicles. By default, all those databases are stored in Hypersonic (HSQLDB). The script for creating the databases in HSQLDB can be found at http://sourceforge.net/projects/pentaho/ files. Under Business Intelligence Server | 1.7.1-stable look for pentaho_sample_data- 1.7.1.zip. It can also be found at svn://ci.pentaho.com/view/Platform/job/bi-platform- sample-data/.
- 28. Chapter 1 7 These databases can be stored in other DBMS as well. Scripts for creating and loading these databases in other popular DBMS as for example MySQL or Oracle can be found in Prashant Raju's blog, at http://www.prashantraju.com/projects/pentaho/. Beside the scripts, you will find instructions for creating and loading the databases. Prashant Raju, an expert Pentaho developer, provides several excellent tutorials related to the Pentaho platform. If you are interested in knowing more about Pentaho, it's worth taking a look at his blog. Connecting to a database If you intend to work with a database, either reading, writing, looking up data, and so on, the first thing you will have to do is to create a connection to that database. This recipe will teach you how to do this. Getting ready In order to create the connection, you will need to know the connection settings. At least you will need: f Host Name: Domain name or IP address of the database server. f Port Number f User Name f Password It's recommended that you also have access to the database at the moment of creating the connection. How to do it... 1. Open Spoon and create a new transformation. 2. Select the View option that appears in the upper-left corner of the screen, right-click the Database connections option, and select New. The Database Connection dialog window appears. 3. Under Connection Type, select the database engine that matches your DBMS.
- 29. Working with Databases 8 4. Fill the Settings options and give the connection a name by typing it in the Connection Name: textbox. Your window should look like this: 5. Press the Test button. A message should appear informing you that the connection to your database is OK. If you get an error message instead, you should recheck the data entered, as well as the availability of the database server. The server might be down, or it might not be reachable from your machine. How it works... A database connection is the definition that allows you to access a database from Kettle. With the data you provide, Kettle can instantiate real database connections and perform the different operations related with databases. Once you define a database connection, you will be able to access that database and execute arbitrary SQL statements: create schema objects like tables, execute SELECT statements, modify rows, and so on. In this recipe you created the connection from the Database connections tree. You may also create a connection by pressing the New... button in the configuration window of any database-related step in a transformation or job entry in a job. Alternatively, there is also a wizard accessible from the Tools menu or by pressing F3. Whichever the method you choose, a setting window like the one you saw in the recipe shows up allowing you to define the connection. This task includes: f Selecting a database engine (Connection type:) f Selecting the access method (Access:)
- 30. Chapter 1 9 Native (JDBC) is recommended but you can also use a predefined ODBC data source, a JNDI data source, or an Oracle OCI connection. f Providing the Host Name or IP f Entering the User Name and Password for accessing the database. A database connection can only be created with a transformation or a job opened. Therefore, in the recipe you were asked to create a transformation. The same could have been achieved by creating a job instead. There's more... The recipe showed the simplest way to create a database connection. However, there is more to know about creating database connections. Avoiding creating the same database connection over and over again If you intend to use the same database in more than one transformation and/or job, it's recommended that you share the connection. You do this by right-clicking the database connection under the Database connections tree, and clicking on Share. This way the database connection will be available to be used in all transformations and jobs. Shared database connections are recognized because they are bold. As an example take a look at the following sample screenshot: The databases books and sampledata are shared; the others are not.
- 31. Working with Databases 10 The information about shared connections is saved in a file named shared.xml located in the Kettle home directory. No matter the Kettle storage method (repository or files) you can share connections. If you are working with the file method, namely ktr and kjb files, the information about shared connections are not only saved in the shared.xml file, but also saved as part of the transformation or job files even if they don't use the connections. You can avoid saving all the connection data as part of your transformations and jobs by selecting the option Only save used connections to XML? in the Kettle options window. Avoiding modifying jobs and transformations every time a connection changes Instead of typing fixed values in the database connection definition, it's worth using variables. For example, instead of typing localhost as the hostname, you can define a variable named HOST_NAME and as host name type its variable notation as ${HOST_NAME} or %%HOST_ NAME%%. If you decide to move the database from the local machine to a server, you just have to change the value of the variable and don't need to modify the transformations or jobs that use the connection. This is especially useful when it's time to move your jobs and transformations between different environments: development, test, and so on. Specifying advanced connection properties The recipe showed you how to provide the general properties needed to create a connection. You may need to specify additional options—for example a preferred schema name, or supply some parameters to be used when the connection is initialized. In order to do that, look for those options in the extra tab windows under the General tab of the Database Connection window. Connecting to a database not supported by Kettle Kettle offers built-in support for a vast set of database engines. The list includes both commercial databases (such as Oracle) and open source (such as PostgreSQL), traditional row-oriented databases (such as MS SQL Server) and modern column-oriented databases (such as Infobright), disk-storage based databases (such as Informix) and in-memory databases (such as HSQLDB). However, it can happen that you want to connect to a database that is not in that list. In that case, you might still create a connection to that database. First of all, you have to get a JDBC driver for that DBMS. Copy the jar file containing the driver to the libext/JDBC directory inside the Kettle installation directory. Then, create the connection. In this case, as connection type choose Generic database. In the Settings frame specify the connection string (which should be explained along with JDBC), the driver class name, and the username and password. In order to find the values for these settings, you will have to refer to the driver documentation.
- 32. Chapter 1 11 Checking the database connection at run-time If you are not sure that the database connection will be accessible when a job or transformation runs from outside Spoon, you might precede all database-related operations with a Check Db connection job entry. The entry will return true or false depending on the result of checking one or more connections. Getting data from a database If you're used to working with databases, one of your main objectives while working with PDI must be getting data from your databases for transforming, loading in other databases, generating reports, and so on. Whatever operation you intend to achieve, the first thing you have to do after connecting to the database, is to get that data and create a PDI dataset. In this recipe you will learn the simplest way to do that. Getting ready To follow these instructions you need to have access to any DBMS. How to do it... 1. Create a transformation and drop into the canvas a Table Input step. You will find it in the Input category of steps. 2. From the Connection drop-down list select the connection to the database where your data resides, or create it if it doesn't exist. 3. In the SQL text area, type the SQL statement that returns the data you need. So far you should have something like this:
- 33. Working with Databases 12 4. Click on Preview. This will bring a sample list of rows so you can confirm that the data is as expected. 5. Press OK to close the Table Input configuration window, and you'll be ready to use the data for further manipulation. How it works... The Table Input step you used in the recipe is the main PDI step to get data from a database. When you run or preview the transformation, Kettle executes the SQL and pushes the rows of data coming from the database into the output stream of the step. Each column of the SQL statement leads to a PDI field and each row generated by the execution of the statement becomes a row in the PDI dataset. Once you get the data from the database, it will be available for any kind of manipulation inside the transformation. There's more... In order to save time, or in case you are not sure of the name of the tables or columns in the database, instead of typing the SQL statement press the Get SQL select statement... button. This will bring the Database Explorer window. This window allows you to explore the selected database. By expanding the database tree and selecting the table that interests you, you will be able to explore that table through the different options available under the Actions menu as shown below: Double-clicking the name of the table will generate a SELECT statement to query that table. You will have the chance to include all the field names in the statement, or simply generate a SELECT * statement. After bringing the SQL to the Table Input configuration window, you will be able to modify it according to your needs.
- 34. Chapter 1 13 By generating this statement you will loose any statement already in the SQL text area. See also f Connecting to a database. In order to get data from a database, you need to have a connection to it. This recipe explains how to do this. f Getting data from a database by providing parameters. This recipe explains a more flexible way to run database queries. f Getting data from a database by running a query built at runtime. This recipe explains an even more flexible method. Getting data from a database by providing parameters If you need to create a dataset with data coming from a database you can do it just by using a Table Input step. If the SELECT statement that retrieves the data doesn't need parameters, you simply write it in the Table Input setting window and proceed. However, most of the times you need flexible queries; queries that receive parameters. This recipe will show you how to pass parameters to a SELECT statement in PDI. Assume that you need to list all products in Steel Wheels for a given product line and scale. Getting ready Make sure you have access to the sampledata database. How to do it... 1. Create a transformation. 2. Before getting the data from the database, you have to create the stream that will provide the parameters for the statement. 3. Create a stream that builds a dataset with a single row and two columns: the product line parameter and the scale parameter. For this exercise, you can do it just by adding a Data Grid step or a Generate Rows step. Doing a preview on the last step of your stream you should see something like this:
- 35. Working with Databases 14 4. Now drag to the canvas a Table Input step, and create a hop from the last step of the stream created above, towards this step. 5. Now you can configure the Table Input step. Double-click it, select the connection to the database, and type the following statement: SELECT productline , productscale , productcode , productname FROM products p WHERE productline = ? AND productscale = ? Downloading the example code You can download the example code fles for all Packt books you have purchased from your account at http://www.PacktPub.com. If you purchased this book elsewhere, you can visit http://www.PacktPub. com/support and register to have the fles e-mailed directly to you. 6. In the Insert data from step list, select the name of the step that is linked to the Table Input step. Close the window. 7. Select the Table Input step and do a preview of the transformation. You will see a list of all products that match the product line and scale provided in the incoming stream: How it works... When you need to execute a SELECT statement with parameters, the first thing you have to do is to build a stream that provides the parameter values needed by the statement. The stream can be made of just one step; for example a Data grid with fixed values, or a stream made up of several steps. The important thing is that the last step delivers the proper values to the Table Input step. Then, you have to link the last step in the stream to the Table Input step where you will type the statement. What differentiates this statement from a regular statement is that you have to provide question marks. When you preview or run the transformation, the statement is prepared and the values coming to the Table Input step are bound to the placeholders; that is, the place where you typed the question marks.
- 36. Chapter 1 15 Note that in the recipe the output of the stream was a single row with two fields, which is exactly the same number of question marks in the statement. The number of fields coming to a Table Input must be exactly the same as the number of question marks found in the query. Also note that in the stream the product line was in the first place and the scale in the sec- ond place. If you look at the highlighted lines in the recipe, you will see that the statement expected the parameter values exactly in that order. The replacement of the markers respects the order of the incoming fields. Finally, it's important to note that question marks can only be used to parameterize value expressions just as you did in the recipe. Keywords or identifiers (for example; table names) cannot be parameterized with the question marks method. If you need to parameterize something different from a value expression you should take another approach as explained in the next recipe. There's more... There are a couple of situations worth discussing. Parameters coming in more than one row In the recipe you received the list of parameter values in a single row with as many columns as expected parameter values. It's also possible to receive the parameter values in several rows. If instead of a row you had one parameter by row, as shown here: The behavior of the transformation wouldn't have changed. The statement would have pulled the values for the two parameters from the incoming stream in the same order as the data appeared. It would have bound the first question mark with the value in the first row, and the second question mark with the value coming in the second row. Note that this approach is less flexible than the previous one. For example, if you have to provide values for parameters with different data types you will not be able to put them in the same column and different rows.
- 37. Working with Databases 16 Executing the SELECT statement several times, each for a different set of parameters Suppose that you not only want to list the Classic Cars in 1:10 scale, but also the Motorcycles in 1:10 and 1:12 scale. You don't have to run the transformation three times in order to do this. You can have a dataset with three rows, one for each set of parameters, as shown below: Then, in the Table Input setting window you have to check the Execute for each row? option. This way, the statement will be prepared and the values coming to the Table Input step will be bound to the placeholders once for each row in the dataset coming to the step. For this example, the result would look like this: See also f Getting data from a database by running a query built at runtime. This recipe gives you an alternative way for using parameters in a SELECT statement. Getting data from a database by running a query built at runtime When you work with databases, most of the times you start by writing an SQL statement that gets the data you need. However, there are situations in which you don't know that statement exactly. Maybe the name of the columns to query are in a file, or the name of the columns by which you will sort will come as a parameter from outside the transformation, or the name of the main table to query changes depending on the data stored in it (for example sales2010). PDI allows you to have any part of the SQL statement as a variable so you don't need to know the literal SQL statement text at design time. Assume the following situation: You have a database with data about books and their authors, and you want to generate a file with a list of titles. Whether to retrieve the data ordered by title or by genre is a choice that you want to postpone until the moment you execute the transformation.
- 38. Chapter 1 17 Getting ready You will need a book database with the structure explained in Appendix, Data Structures. How to do it... 1. Create a transformation. 2. The column that will define the order of the rows will be a named parameter. So, define a named parameter named ORDER_COLUMN, and put title as its default value. Remember that Named Parameters are defined in the Transformation setting window and their role is the same as the role of any Kettle variable. If you prefer, you can skip this step and define a standard variable for this purpose. 3. Now drag a Table Input step to the canvas. Then create and select the connection to the book's database. 4. In the SQL frame type the following statement: SELECT * FROM books ORDER BY ${ORDER_COLUMN} 5. Check the option Replace variables in script? and close the window. 6. Use an Output step like for example a Text file output step, to send the results to a file, save the transformation, and run it. 7. Open the generated file and you will see the books ordered by title. 8. Now try again. Press F9 to run the transformation one more time. 9. This time, change the value of the ORDER_COLUMN parameter typing genre as the new value. 10. Press the Launch button. 11. Open the generated file. This time you will see the titles ordered by genre. How it works... You can use Kettle variables in any part of the SELECT statement inside a Table Input step. When the transformation is initialized, PDI replaces the variables by their values provided that the Replace variables in script? option is checked. In the recipe, the first time you ran the transformation, Kettle replaced the variable ORDER_ COLUMN with the word title and the statement executed was: SELECT * FROM books ORDER BY title
- 39. Working with Databases 18 The second time, the variable was replaced by genre and the executed statement was: SELECT * FROM books ORDER BY genre As mentioned in the recipe, any predefined Kettle variable can be used instead of a named parameter. There's more... You may use variables not only for the ORDER BY clause, but in any part of the statement: table names, columns, and so on. You could even hold the full statement in a variable. Note however that you need to be cautious when implementing this. A wrong assumption about the metadata generated by those predefined statements can make your transformation crash. You can also use the same variable more than once in the same statement. This is an advantage of using variables as an alternative to question marks when you need to execute parameterized SELECT statements. See also f Getting data from a database by providing parameters. This recipe shows you an alternative way to parameterize a query. Inserting or updating rows in a table Two of the most common operations on databases besides retrieving data are inserting and updating rows in a table. PDI has several steps that allow you to perform these operations. In this recipe you will learn to use the Insert/Update step. Before inserting or updating rows in a table by using this step, it is critical that you know which field or fields in the table uniquely identify a row in the table. If you don't have a way to uniquely identify the records, you should consider other steps, as explained in the There's more... section. Assume this situation: You have a file with new employees of Steel Wheels. You have to insert those employees in the database. The file also contains old employees that have changed either the office where they work, or the extension number, or other basic information. You will take the opportunity to update that information as well.
- 40. Chapter 1 19 Getting ready Download the material for the recipe from the book's site. Take a look at the file you will use: NUMBER, LASTNAME, FIRSTNAME, EXT, OFFICE, REPORTS, TITLE 1188, Firrelli, Julianne,x2174,2,1143, Sales Manager 1619, King, Tom,x103,6,1088,Sales Rep 1810, Lundberg, Anna,x910,2,1143,Sales Rep 1811, Schulz, Chris,x951,2,1143,Sales Rep Explore the Steel Wheels database, in particular the employees table, so you know what you have before running the transformation. In particular execute these statements: SELECT EMPLOYEENUMBER ENUM , concat(FIRSTNAME,' ',LASTNAME) NAME , EXTENSION EXT , OFFICECODE OFF , REPORTSTO REPTO , JOBTITLE FROM employees WHERE EMPLOYEENUMBER IN (1188, 1619, 1810, 1811); +------+----------------+-------+-----+-------+-----------+ | ENUM | NAME | EXT | OFF | REPTO | JOBTITLE | +------+----------------+-------+-----+-------+-----------+ | 1188 | Julie Firrelli | x2173 | 2 | 1143 | Sales Rep | | 1619 | Tom King | x103 | 6 | 1088 | Sales Rep | +------+----------------+-------+-----+-------+-----------+ 2 rows in set (0.00 sec) How to do it... 1. Create a transformation and use a Text File input step to read the file employees. txt. Provide the name and location of the file, specify comma as the separator, and fill in the Fields grid. Remember that you can quickly fill the grid by pressing the Get Fields button. 2. Now, you will do the inserts and updates with an Insert/Update step. So, expand the Output category of steps, look for the Insert/Update step, drag it to the canvas, and create a hop from the Text File input step toward this one. 3. Double-click the Insert/Update step and select the connection to the Steel Wheels database, or create it if it doesn't exist. As target table type employees.
- 41. Working with Databases 20 4. Fill the grids as shown: 5. Save and run the transformation. 6. Explore the employees table. You will see that one employee was updated, two were inserted, and one remained untouched because the file had the same data as the database for that employee: +------+---------------+-------+-----+-------+--------------+ | ENUM | NAME | EXT | OFF | REPTO | JOBTITLE | +------+---------------+-------+-----+-------+--------------+ | 1188 | Julie Firrelli| x2174 | 2 | 1143 |Sales Manager | | 1619 | Tom King | x103 | 6 | 1088 |Sales Rep | | 1810 | Anna Lundberg | x910 | 2 | 1143 |Sales Rep | | 1811 | Chris Schulz | x951 | 2 | 1143 |Sales Rep | +------+---------------+-------+-----+-------+--------------+ 4 rows in set (0.00 sec) How it works... The Insert/Update step, as its name implies, serves for both inserting or updating rows. For each row in your stream, Kettle looks for a row in the table that matches the condition you put in the upper grid, the grid labeled The key(s) to look up the value(s):. Take for example the last row in your input file: 1811, Schulz, Chris,x951,2,1143,Sales Rep When this row comes to the Insert/Update step, Kettle looks for a row where EMPLOYEENUMBER equals 1811. It doesn't find one. Consequently, it inserts a row following the directions you put in the lower grid. For this sample row, the equivalent INSERT statement would be: INSERT INTO employees (EMPLOYEENUMBER, LASTNAME, FIRSTNAME, EXTENSION, OFFICECODE, REPORTSTO, JOBTITLE) VALUES (1811, 'Schulz', 'Chris', 'x951', 2, 1143, 'Sales Rep')
- 42. Chapter 1 21 Now look at the first row: 1188, Firrelli, Julianne,x2174,2,1143, Sales Manager When Kettle looks for a row with EMPLOYEENUMBER equal to 1188, it finds it. Then, it updates that row according to what you put in the lower grid. It only updates the columns where you put Y under the Update column. For this sample row, the equivalent UPDATE statement would be: UPDATE employees SET EXTENSION = 'x2174' , OFFICECODE = 2 , REPORTSTO = 1143 , JOBTITLE = 'Sales Manager' WHERE EMPLOYEENUMBER = 1188 Note that the name of this employee in the file (Julianne) is different from the name in the table (Julie), but, as you put N under the column Update for the field FIRSTNAME, this column was not updated. If you run the transformation with log level Detailed, you will be able to see in the log the real prepared statements that Kettle performs when inserting or updating rows in a table. There's more... Here there are two alternative solutions to this use case. Alternative solution if you just want to insert records If you just want to insert records, you shouldn't use the Insert/Update step but the Table Output step. This would be faster because you would be avoiding unnecessary lookup operations. The Table Output step is really simply to configure: Just select the database connection and the table where you want to insert the records. If the names of the fields coming to the Table Output step have the same name as the columns in the table, you are done. If not, you should check the Specify database fields option, and fill the Database fields tab exactly as you filled the lower grid in the Insert/Update step, except that here there is no Update column. Alternative solution if you just want to update rows If you just want to update rows, instead of using the Insert/Update step, you should use the Update step. You configure the Update step just as you configure the Insert/Update step, except that here there is no Update column.
- 43. Working with Databases 22 Alternative way for inserting and updating The following is an alternative way for inserting and updating rows in a table. This alternative only works if the columns in the Key field's grid of the Insert/ Update step are a unique key in the database. You may replace the Insert/Update step by a Table Output step and, as the error handling stream coming out of the Table Output step, put an Update step. In order to handle the error when creating the hop from the Table Output step towards the Update step, select the Error handling of step option. Alternatively right-click the Table Output step, select Define error handling..., and configure the Step error handling settings window that shows up. Your transformation would look like this: In the Table Output select the table employees, check the Specify database fields option, and fill the Database fields tab just as you filled the lower grid in the Insert/Update step, excepting that here there is no Update column. In the Update step, select the same table and fill the upper grid—let's call it the Key fields grid—just as you filled the key fields grid in the Insert/Update step. Finally, fill the lower grid with those fields that you want to update, that is, those rows that had Y under the Update column. In this case, Kettle tries to insert all records coming to the Table Output step. The rows for which the insert fails go to the Update step, and the rows are updated. If the columns in the Key field's grid of the Insert/Update step are not a unique key in the database, this alternative approach doesn't work. The Table Output would insert all the rows. Those that already existed would be duplicated instead of updated. This strategy for performing inserts and updates has been proven to be much faster than the use of the Insert/Update step whenever the ratio of updates to inserts is low. In general, for best practices reasons, this is not an advisable solution.
- 44. Chapter 1 23 See also f Inserting new rows when a simple primary key has to be generated. If the table where you have to insert data defines a primary key, you should generate it. This recipe explains how to do it when the primary key is a simple sequence. f Inserting new rows when the primary key has to be generated based on stored values. Same as the previous bullet, but in this case the primary key is based on stored values. Inserting new rows where a simple primary key has to be generated It's very common to have tables in a database where the values for the primary key column can be generated by using a database sequence (in those DBMS that have that feature; for example, Oracle) or simply by adding 1 to the maximum value in the table. Loading data into these tables is very simple. This recipe teaches you how to do this through the following exercise. There are new offices at Steel Wheels. You have the data of the offices in a file that looks like this: CITY;PHONE;ADDRESS;COUNTRY;POSTALCODE Sao Paulo;11 3289-3703;Avenida Paulista 1330;Brazil;01310-200 Sao Paulo;11 3104-1116;Rua Boa Vista, 51;Brazil;01014-001 Buenos Aires;11 4582-6700;Cabildo 2127;Argentina;C1428AAT You have to insert that data into the Steel Wheels database. Getting ready For this recipe you will use the Pentaho sample database. If you don't have that database, you'll have to follow the instructions in the introduction of this chapter. As you will insert records into the office table, it would be good if you explore that table before doing any insert operations. Here you have a sample query: SELECT OFFICECODE, ADDRESSLINE1, CITY FROM OFFICES ORDER BY OFFICECODE; +------------+--------------------------+---------------+ | OFFICECODE | ADDRESSLINE1 | CITY | +------------+--------------------------+---------------+ | 1 | 100 Market Street | San Francisco | | 2 | 1550 Court Place | Boston | | 3 | 523 East 53rd Street | NYC |
- 45. Working with Databases 24 | 4 | 43 Rue Jouffroy D'abbans | Paris | | 5 | 4-1 Kioicho | Tokyo | | 6 | 5-11 Wentworth Avenue | Sydney | | 7 | 25 Old Broad Street | London | +------------+--------------------------+---------------+ 7 rows in set (0.00 sec) How to do it... 1. Create a transformation and create a connection to the sampledata database. 2. Use a Text file input step to read the offices.txt file with data about the new offices. 3. From the Data Warehouse category drag and drop a Combination lookup/update step, and create a hop from the previous step towards this one. 4. Double-click the step, select the connection to the sampledata database, and type offices as the Target table. 5. Fill the Key fields grid as shown: 6. In the Technical key field type OFFICECODE. For the Creation of technical key fields leave the default values. Close the window. 7. From the Output category of steps, add an Update step. 8. Double-click the step, select the connection to the sampledata database, and type offices as the Target table. 9. In the first grid add rows with the text OFFICECODE both under Table field and under Stream field1. As Comparator choose =. This way, you will update the rows where OFFICECODE is equal to the office code in your stream. 10. In the lower grid add a row and type PHONE both under Table field and Stream field. Add a second row and type POSTALCODE in both columns. 11. Close the window. 12. It's time to save the transformation and run it to see what happens. 13. As you might guess, three new offices have been added, with primary keys 8, 9, and 10. Look at the results: SELECT OFFICECODE, ADDRESSLINE1, CITY FROM offices ORDER BY cast(officecode as unsigned);
- 46. Chapter 1 25 +------------+--------------------------+---------------+ | OFFICECODE | ADDRESSLINE1 | CITY | +------------+--------------------------+---------------+ | 1 | 100 Market Street | San Francisco | | 2 | 1550 Court Place | Boston | | 3 | 523 East 53rd Street | NYC | | 4 | 43 Rue Jouffroy D'abbans | Paris | | 5 | 4-1 Kioicho | Tokyo | | 6 | 5-11 Wentworth Avenue | Sydney | | 7 | 25 Old Broad Street | London | | 8 | Avenida Paulista 1330 | Sao Paulo | | 9 | Rua Boa Vista, 51 | Sao Paulo | | 10 | Cabildo 2127 | Buenos Aires | +------------+--------------------------+---------------+ 10 rows in set (0.00 sec) How it works... In many situations, before inserting data into a table you have to generate the primary key. If the primary key is a simple sequence or the maximum primary key plus one, you can generate it by using a Combination lookup/update step. In the recipe, for each row in your file, with the Combination lookup/update step, you look for a record in the offices table with the same values for address, city, and country. Because the offices are new, (there aren't offices in the table with the same combination of address, city, and country values) the lookup fails. As a consequence, the step generates a key value as the maximum OFFICECODE in the table, plus 1. Then, it inserts a row with the generated primary key and the fields you typed in the grid. Finally, the step adds to the stream the generated primary key value. As a last task, we used that key to update the other fields coming into the file: POSTALCODE and PHONE. There's more... The Combination lookup/update step is within the Data Warehouse category, because is mainly used for loading junk dimension tables. But as you could see, it can also be used in the particular situation where you have to generate a primary key. In the recipe you generated the PK as the maximum plus one, but as you can see in the setting window, a database sequence can also be used instead.
- 47. Working with Databases 26 When you use the Combination lookup/update step for inserting, make sure that the columns that are not part of the list of key fields are nullable or have default values. Using the Combination lookup/update for looking up In the recipe the Combination lookup/update step just inserted new rows. Now suppose that you have a row that existed in the table. In that case the lookup would have succeeded and the step wouldn't have inserted a new row. Instead, it would just have returned the found OFFICECODE. That field would have been added to the stream, ready to be used further in the transformation, for example for updating other fields as you did in the recipe, or for being used for inserting data in a related table. Note that this is a potentially slow step, as it uses all the values for the comparison. See also f Inserting new rows when the primary key has to be generated based on stored values. This recipe explains the case where the primary key to be generated is not as simple as adding one to the last primary key in the table. Inserting new rows where the primary key has to be generated based on stored values There are tables where the primary key is not a database sequence, nor a consecutive integer, but a column which is built based on a rule or pattern that depends on the keys already inserted. For example imagine a table where the values for primary key are A00001, A00002, and A00003. In this case, you can guess the rule: putting an A followed by a sequence. The next in the sequence would be A00004. This seems too simple, but doing it in PDI is not trivial. This recipe will teach you how to load a table where a primary key has to be generated based on existing rows as in that example. Suppose that you have to load author data into the book's database. You have the main data for the authors, and you have to generate the primary key as in the example above. Getting ready Run the script that creates and loads data into the book's database. You'll find it at http://packtpub.com/support.
- 48. Chapter 1 27 Before proceeding, verify the current values for the primary keys in the table where you will insert data: SELECT MAX(id_author) FROM authors; +----------------+ | MAX(id_author) | +----------------+ | A00009 | +----------------+ 1 row in set (0.00 sec) How to do it... 1. Create a transformation and create a connection to the book's database. 2. Use a Text file input step to read the authors.txt file. For simplicity, the authors.txt file only has new authors, that is, authors who are not in the table. 3. To generate the next primary key, you need to know the current maximum. So use a Table Input step to get it. In this case the following statement will give you that number: SELECT cast(max(right(id_author,5)) as unsigned) max_id FROM authors Alternatively you can simply get the id_author field and transform the field with Kettle steps until you get the current maximum. You will have a simple clear transformation, but it will take several Kettle steps to do it. 4. By using a Join Rows (cartesian product) step, join both streams. Your transformation should look like this:
- 49. Working with Databases 28 5. Add an Add sequence step. Replace the default value valuename with delta_ value. For the rest of the fields in the setting window leave the default values. 6. Add a Calculator step to build the keys. You do it by filling the setting window as shown: 7. In order to insert the rows, add a Table output step, double-click it, and select the connection to the book's database. 8. As Target table type authors. 9. Check the option Specify database fields. 10. Select the Database fields tab and fill the grid as follows: 11. Save and run the transformation. 12. Explore the authors table. You should see the new authors: SELECT * FROM authors ORDER BY id_author; +----------+-----------+-------------+-----------+----------+ | lastname | firstname | nationality | birthyear | id_author| +----------+-----------+-------------+-----------+----------+ | Larsson | Stieg | Swedish | 1954 | A00001 | | King | Stephen | American | 1947 | A00002 | | Hiaasen | Carl | American | 1953 | A00003 | | Handler | Chelsea | American | 1975 | A00004 | | Ingraham | Laura | American | 1964 | A00005 | | Ramsey | Dave | American | 1960 | A00006 | | Kiyosaki | Robert | American | 1947 | A00007 | | Rowling | Joanne | English | 1965 | A00008 | | Riordan | Rick | American | 1964 | A00009 | | Gilbert | Elizabeth | unknown | 1900 | A00010 | | Franzen | Jonathan | unknown | 1900 | A00011 | | Collins | Suzanne | unknown | 1900 | A00012 | | Blair | Tony | unknown | 1900 | A00013 |
- 50. Chapter 1 29 +----------+-----------+-------------+-----------+----------+ 13 rows in set (0.00 sec) How it works... When you have to generate a primary key based on the existing primary keys, unless the new primary key is simple to generate by adding one to the maximum, there is no direct way to do it in Kettle. One possible solution is the one shown in the recipe: Getting the last primary key in the table, combining it with your main stream, and using those two sources for generating the new primary keys. This is how it worked in this example. First, by using a Table Input step, you found out the last primary key in the table. In fact, you got only the numeric part needed to build the new key. In this exercise, the value was 9. With the Join Rows (cartesian product) step, you added that value as a new column in your main stream. Taking that number as a starting point, you needed to build the new primary keys as A00010, A00011, and so on. You did this by generating a sequence (1, 2, 3, and so on), adding this sequence to the max_id (that led to values 10, 11, 12, and so on), and finally formatting the key with the use of the calculator. Note that in the calculator the first A+B performs an arithmetic calculation. It adds the max_id with the delta_value sequence. Then it converts the result to a String giving it the format with the mask 0000. This led to the values 00010, 00011, and so on. The second A+B is a string concatenation. It concatenates the literal A with the previously calculated ID. Note that this approach works as long as you have a single user scenario. If you run multiple instances of the transformation they can select the same maximum value, and try to insert rows with the same PK leading to a primary key constraint violation. There's more... The key in this exercise is to get the last or maximum primary key in the table, join it to your main stream, and use that data to build the new key. After the join, the mechanism for building the final key would depend on your particular case. See also f Inserting new rows when a simple primary key has to be generated. If the primary key to be generated is simply a sequence, it is recommended to examine this recipe.
- 51. Working with Databases 30 Deleting data from a table Sometimes you might have to delete data from a table. If the operation to do is simple, for example: DELETE FROM LOG_TABLE WHERE VALID='N' Or DELETE FROM TMP_TABLE You could simply execute it by using an SQL job entry or an Execute SQL script step. If you face the second of the above situations, you can even use a Truncate table job entry. For more complex situations you should use the Delete step. Let's suppose the following situation: You have a database with outdoor products. Each product belongs to a category: tools, tents, sleeping bags, and so on. Now you want to delete all the products for a given list of categories, where the price is less than or equal to $50. Getting ready In order to follow the recipe, you should download the material for this chapter: a script for creating and loading the database, and an Excel file with the list of categories involved. After creating the outdoor database and loading data by running the script provided, and before following the recipe you can explore the database. In particular execute the following statement: SELECT category, count(*) quantity FROM products p, categories c WHERE p.id_category=c.id_category AND price<=50 GROUP BY p.id_category; +---------------+----------+ | category | quantity | +---------------+----------+ | kitchen | 19 | | lights | 14 | | sleeping bags | 5 | | tents | 4 | | tools | 8 | +---------------+----------+ 5 rows in set (0.00 sec) SELECT category, count(*) quantity FROM products p, categories c
- 52. Chapter 1 31 WHERE p.id_category=c.id_category AND price>50 GROUP BY p.id_category; +---------------+----------+ | category | quantity | +---------------+----------+ | kitchen | 5 | | lights | 1 | | sleeping bags | 1 | | tents | 8 | | tools | 2 | +---------------+----------+ 5 rows in set (0.00 sec) The highlighted lines above belong to the products that you intend to delete. How to do it... 1. Create a transformation. 2. The value to which you will compare the price before deleting will be stored as a named parameter. So create it, name it MAX_PRICE, and set 50 as the default value. 3. Drag to the canvas an Excel Input step to read the Excel file with the list of categories. 4. Drag to the canvas a Get Variables step, to get the named variable as a field named max_price with type Number. 5. After that, add a Database lookup step. Configure it to get the id_category fields based on the category descriptions in the Excel file. So far, the transformation looks like this: For higher volumes it's better to get the variable just once in a separate stream and join the two streams with a Join Rows (cartesian product) step. 6. Select the Database lookup step and do a preview. You should see this:
- 53. Working with Databases 32 7. Finally, add a Delete step. You will find it under the Output category of steps. 8. Double-click the Delete step, select the outdoor connection, and fill in the key grid as follows: 9. Save and run the transformation. 10. Explore the database. If you run the same statements that you ran before starting the recipe, you'll note that all products belonging to the categories in the Excel file, with price less than or equal to $50 have been deleted. This is what you will see: SELECT category, count(*) quantity FROM products p, categories c WHERE p.id_category=c.id_category AND price<=50 GROUP BY p.id_category; +---------------+----------+ | category | quantity | +---------------+----------+ | kitchen | 19 | | lights | 14 | | sleeping bags | 5 | +---------------+----------+ 3 rows in set (0.00 sec) SELECT category, count(*) quantity FROM products p, categories c WHERE p.id_category=c.id_category AND price>50 GROUP BY p.id_category; +---------------+----------+ | category | quantity | +---------------+----------+ | kitchen | 5 | | lights | 1 | | sleeping bags | 1 | | tents | 8 | | tools | 2 | +---------------+----------+ 5 rows in set (0.00 sec)
- 54. Chapter 1 33 How it works... The Delete step allows you to delete rows in a table in a database based on certain conditions. In this case, you intended to delete rows from the table products where the price was less than or equal to 50, and the category was in a list of categories, so the Delete step is the right choice. This is how it works. PDI builds a prepared statement for the DELETE operation. Then, for each row in your stream, PDI binds the values of the row to the variables in the prepared statement. Let's see it by example. In the transformation you built a stream where each row had a single category and the value for the price. If you run the transformation with log level Detailed and look at the log, you will see the statement that is executed: DELETE FROM products WHERE price < ? AND id_category = ? The WHERE clause is built based on the conditions you entered in the Delete configuration window. For every row, the values of the fields you typed in the grid—max_price and id_ category—are bound to the question marks in the prepared statement. Note that the conditions in the Delete step are based on fields in the same table. In this case, as you were provided with category descriptions and the products table does not have the descriptions but the ID for the categories, you had to use an extra step to get that ID: a Database lookup. Suppose that the first row in the Excel file had the value tents. As the ID for the category tents is 4, the execution of the prepared statement with the values in this row has the same effect as the execution of the following SQL statement: DELETE FROM products WHERE price < 50 AND id_category = 4 See also f Looking for a value in a database table (Chapter 5, Looking for Data). Refer to this recipe if you need to understand how the Database lookup step works.
- 55. Other documents randomly have different content
- 56. On January 26th, 1916, Sir Edward Grey delivered his promised reply in the House of Commons. It was brilliant oratory, but it was not argument. It was a defence of the Navy, which needed no defence. It was a masterpiece of forensic jurisprudence, but it revealed between the chinks of polished sentences and high-sounding declamation, in startling nakedness, the weaknesses, the unwarrantable hesitating caution, or the downright cowardice of the Cabinet. With such grace and skill did the speaker unfold his case that a reader, unaware of the facts concealed behind it, would believe the policy and actions of the Government had been hitherto faultless, flawless, and blameless. Reading it at a later date brought to my mind the story of a poacher's wife, who with tears of grateful joy streaming down her countenance, thanked a learned junior counsel for his able and successful defence of her husband, who had been charged with stealing a certain shot- gun. "My good woman," replied her modest advocate, "it was only a mistake. The judge truly said that your good husband left the Court without a stain upon his character. It was only alleged that he stole the gun." "Alleged be bothered," said the woman; "why, we've got the gun at home now!" If this speech of Sir Edward Grey, as a speech, had a fault at all, it was that the defence he made was too good to ring true. At the time of its utterance it appeared to appease the House. No one wished to hamper the Government, which, like the energetic but painfully inefficient pianist at a certain Western mining camp, was protected by proclamation: "Please don't shoot. He's doing his best." But outside the House the underlying effect of the speech upon thinking people was very different. It created satisfaction in Germany and amongst neutral Governments. It caused great jubilation amongst the vast army of mushroom traders and adventurers abroad who were piling up fortunes by illicit trading. But it left Englishmen and our true sympathisers in this tragic war irritable, indignant, and unsatisfied; smouldering in their just wrath at the confessed weak-kneed policy of
- 57. politicians, who, however good their intentions, proved that they had not yet grasped the difference between a quarrel at law and a quarrel at war. It left the nation disappointed. The people felt we had been fooling with the war too long; that the time had arrived for some strong and decisive action. That politics and patronage should be shelved and the Navy given a free hand. It remembered how the Government had hesitated, procrastinated, and vacillated in this so-called blockade, as in other matters. It remembered that Parliament had refused to pass a code of international rules called the Declaration of London because that code, made largely to please Germany, weakened the hands of the Navy. It remembered that the Government had gone behind the back of Parliament and illegally put that very code into operation after war began. It had not forgotten that this proved such a scandalous weakening of our right and our strength that soon after the Coalition Government came into being that code was said to have been scrapped. Even as to this doubts arose for long afterwards.[21] It had not forgotten the seventeen long months of public pressure and the trouble there had been to force cotton as contraband; nor the seventeen months of "wait and see" before the Navy was permitted to examine mails and extract (inter alia) parcels of rubber. It had not forgotten Sir Edward Grey's declaration that "he had no intention of making cotton contraband"; nor Lord Haldane's contention that "it was useless stopping the import of cotton to Germany, because if we did Germany could find a substitute for it." The nation had been deceived and lulled to sleep before by soft words and gentle assurances. It had been told, "we decline to be bound by judicial niceties." It had been promised "to prevent commodities of any kind from entering or leaving the enemy's country"; "to stick at nothing." It remembered with some misgiving how these promises had been kept.[22] What, it reasoned, were the disappointments of a few Dutch and Scandinavian adventurers from making fortunes out of a war which to
- 58. ourselves was a tragedy? The country had unbounded confidence in the Navy. It had not unbounded confidence in either the Government or the Foreign Office. It hungered with an overwhelming desire to know why the Navy should not be given a free and unhampered hand. The speaker skilfully evaded too much information on that point, and the nation was compelled to nurse its resentment. At the outset of his speech, Sir Edward Grey attempted to deal with the mass of statistics and evidence of direct importation of goods into Germany accumulated by the Press. He selected wheat and flour only, whilst he casually referred to a list of figures issued by the Press Bureau from the War Trade Department of the Government the day before the debate, which members in the House rightly complained had not been supplied to themselves. This list was stated to have been compiled officially in this country from true copies of the ships' manifests, and it alleged the figures given by the Danish Borsen were in many cases wrong and unduly inflated. For instance, the increase in rice imports should have been only 480 per cent. as against 580 per cent.; lard, 275 per cent. instead of 375 per cent.; pork only 1,216 per cent. instead of 1,300 per cent.; and so on. Now everyone knows that statistics are not infallible and a generous allowance should always be made by a careful calculator. But when all circumstances are taken into consideration it can safely be concluded that the majority of the increases alleged by the various Press writers, as having percolated into Germany, were, if anything, under rather than over the mark. As to the reliability of the Borsen, it is edited by a Government statistician, and considered by Danish traders as official. So far as Norway is concerned, H.B.M. Minister at Christiania had difficulty in obtaining official statistics regarding imports and exports after the Casement affair remained unanswered; certain it is that Government assistance was denied to various Consuls acting under him; whilst I, when in that country, was informed (by British
- 59. authorities) I must not collect these figures, although to me and others working with me they were comparatively easy of access. So far as Foreign Office knowledge is concerned, it is hardly a credit to the ability or even sanity of the British Legations in Scandinavia if they have denied knowledge of these colossal imports of goods into Germany, which were known to almost every inhabitant of seaport towns. If they deliberately shut their eyes to the evidence all around them, they presumably obeyed orders. One could then only wonder as to the reason for such suicidal policy. As before mentioned, at the commencement of his speech Sir Edward Grey laid stress upon the fact that part of the stated increased import, namely, 2,000,000 barrels of flour were allowed to be exported to Belgium; whilst a little later in his speech he admitted that "She [Germany] had requisitioned the food supplies of the civil population of Poland and Belgium." Almost immediately afterwards Lord Robert Cecil strove hard to back up the Secretary for Foreign Affairs, but he could not give the House any positive assurance that the Belgian Relief distribution was absolutely independent of German control. The disposition of this is therefore obvious. Sir Edward Grey attempted to whittle down the U.S.A. exports of wheat by stating that nearly half went to Spain, Portugal, Greece, and Malta; but he did not refer to the corn, etc., exported to Northern neutrals from Liverpool and other British ports, nor did he make any allowances for the stream of mysterious ships sailing round far northern seas (many of them choosing the passage north of Iceland), which sighted land on the north-western coast of Norway and carried their course inside neutral waters into the Baltic; which heavily-laden cargo-boats I and others in the Secret Service had watched and reported week by week and month by month with heart-rending persistency. The majority of these ships probably sailed direct to German ports, and no records of their cargoes were likely to be made, or returned from any country concerning them. Nor did Sir Edward Grey make reference to the grain ships, which although nominally bound for Scandinavian ports, were intercepted by their
- 60. owners' or consignees' agents in the Baltic, for the purpose of varying orders for their ultimate port of destination; nor to the ships which were held up in the Baltic by German war vessels and taken to German ports under circumstances calling for grave investigation. Nor did he attempt to answer the general American statistics showing that the gain in imports to northern neutral countries exceeded the German loss. About the middle of his speech Sir Edward Grey said: "If a vessel was held up by the Fleet with suspected cargo on board, the matter was referred to the contraband committee, who decided what part of the cargo should go to the Prize Court." Surely any other nation in the world at war would have arranged from the outset that the capture of a vessel with contraband on board en route for the enemy, would have meant confiscation of the ship and her cargo. Our exceptional and extraordinary leniency was hardly commented upon; it was certainly not satisfactorily explained. Continuing to quote from the speech: He would say to neutrals that we could not give up the right to interfere with enemy trade and must maintain and press that point. He would ask those countries in considering our rights to apply the principles which were applied by the American Government in the war between the North and South as affected by modern conditions. If they agreed to it, then let them with their Chambers of Commerce and other bodies make it easier for us to distinguish between goods intended for the enemy and goods intended for themselves. If those neutral countries said that we were not entitled to prevent trading through, neutral countries with the enemy, then he (Sir E. Grey) must say to the neutral countries who took that line that it was a departure from neutrality. (Cheers.) But he did not think they would take that line. What naturally strikes the reader on perusal is this: why not the words, "I had said" and "I have asked" instead of "he would say" and "he would ask" which Sir Edward Grey used in his speech? Why wait eighteen months to arrive at such a decision? Why were not these
- 61. words used as soon as war was declared? Flagrant breaches arose, as Sir Edward Grey should or must have known, and continued to increase in magnitude from the autumn of 1914. Why he waited until the then date, and why he had not acted before, was not explained. In the next few grandiloquent sentences he admitted the justification and the necessity; whilst the House cheered the words, forgetting past neglected deeds. Next he admitted that "Germany had, in effect, treated food, when she found it, as absolute contraband since the first outbreak of war." This admission gave one much to ponder over. On the point of a stricter blockade Sir Edward Grey suggested that "if a rigorous blockade had been established the whole world would have been against us." Such a contingency, put into legal parlance, is too ridiculously remote for further consideration. Why did he not explain why our Fleet was not allowed to limit particular imports to neutral countries to certain fixed totals per month, or per annum? It is unthinkable to suppose that any country would seriously threaten war in face of former well- known precedent and because such limits were imposed by a blockading Fleet. More particularly so if any such affected country happened to have been one of the parties to the Treaty of the Hague, which affirmed the integrity of poor innocent, unoffending Belgium; the country which, without justification or excuse, was violated, and ravished, outraged by the barbarian Hun invaders, and which so many other countries watched aghast without attempting to help England to protect or to avenge. Admittedly it would have been easy for us to close the Baltic and the Mediterranean. Why did we not do so? We could then have regulated to each country not at war its full and fair average annual complement of necessities plus an extra and a generous margin for contingencies. The Government of each recipient country would have seen to it that its own respective countrymen reaped full benefits; leaks to the Central Powers would have automatically stopped.
- 62. What countries would such a course of action have forced into war against us? Possibly Sweden, doubtfully Holland, remotely Denmark. America had boasted she was "too proud to fight." She might have favoured us with a "note," but her love of trade would have been an absolute bar to the possibility of any cessation of supplies and munitions. No other country would have demurred except Greece, and the vacillating tactics of the Greeks were but the harvest which could have been expected from the seed of "wait-and-see" diplomatic sowing. This is clearly shown by the utterances of King Tino, who said: "I fear the Germans. I do not fear the English." The Greeks have similarly expressed themselves. "We know the Germans would rob, murder, and outrage our land and our people without any hesitation. The English are quite incapable of anything of that kind." It had been proved that Consulates in Greece had been nests of espionage and arsenals of munitions, and the Islands bases for submarine murderers; and yet their King actually sent us a protest against our movement at Salonika to assist the persecuted Serbians whom he and his country had pledged themselves to uphold and protect; a solemn treaty they had long ago undertaken, but so conveniently forgotten and lamely excused themselves out of as soon as called upon to carry it into active force. As a general answer to the direct charges of the Press that the Foreign Office had not kept faith with the nation in doing all that could be done to make an effective blockade, as an explanation to sweep on one side the overwhelming mass of evidence relating to the extraordinary number of German agents and dealers who swarmed throughout Scandinavia and Holland, their amazing advertisements, their suddenly accumulated wealth, the balance sheets showing large profits of neutral companies dealing in Germany's requirements, the alleged wholesale dealers of imported goods so suddenly sprung up from the ranks of hotel porters, clerks, typists, adventurers,
- 63. caretakers, and even charwomen and servant-girls, our own inflated home prices of necessities and commodities—Sir Edward Grey's answer to all this was: The Government had lately sent Lord Faringdon to examine the position in Holland and Scandinavia and he reported that on the whole things were very satisfactory and that all was being done that could be done to prevent the enemy obtaining supplies. Well might the fat stomachs of the "Goulashes"[23] extend and shake in merriment when they read these comfortable words! Sir Edward Grey concluded his speech with this stirring peroration: The whole of our resources were engaged in this war, and our maximum effort was at the disposal of our Allies in carrying on this conflict. With them we should see it through to the end and we should slacken no effort in the common cause. We should exert all our efforts to put the maximum possible pressure upon the enemy, and part of that pressure must be doing the most we could to prevent supplies going to or from the enemy, using the Navy to its full power ... and in common with our Allies sparing nothing, whether it were military, naval, or financial effort, which this country could afford, to see the thing through with them to the end. In the loud cheering with which the House of Commons received the speech no thought was given to the famous words of Napoleon: "Put no faith in talk which is not borne out by action"; whilst future events went to show that Napoleon truly forecasted England's present-day weakness when he wrote: "Feebleness in its Government is the most frightful calamity that can befall a nation." Contrast Sir Edward Grey's eloquent words and diplomatic evasiveness upon the treatment of neutrals with the plain, outspoken, thoroughly English opinion of Lord Fisher, who is credited with having said: "There are no such things as neutral powers. Powers are either with us or against us. If they are friendly they will put up with some
- 64. inconvenience; if they are unfriendly they will squeal. Let them squeal." Had we acted throughout on this dictum the war would most probably have been over well inside of eighteen months. Men of the calibre of this grand old Sea Lord, whose farsight, foresight, and second sight have endeared him to the nation and made him unique and incomparable, would soon have made short work of the war. Yet they were not wanted by the then present-day party-system Government. They were much too blunt and honest and energetically active. The nation will also remember that when Lord Kitchener of Khartoum returned from the East in the early days of the then present Government, it had no use for his invaluable services. He was actually permitted to accept a directorship of one of our poorest railway companies on the south coast for want of a better occupation.[24] But the Press and the public soon brought the Government to book, as it seemingly had to do in every matter of real national importance. The Government tried to keep Lord Haldane installed at the War Office, but the Press would have none of it. It also insisted on K. of K. being placed in his proper place and kept there. More's the pity that he was not given a free hand to do as he liked. The Press also clamoured for Lord Fisher as First Lord of the Admiralty. The nation knows how he was treated. A captain in the Navy aptly described the unwanted and slighted Admiral expert in John Bull, February, 1916, as follows: "Lord John Fisher is to-day our second Nelson—a diplomatist among diplomats and a strategist unequalled in our history. What has Lord John Fisher done? "He scrapped 162 obsolete warships which were rotting in harbour at great expense—for which the Government tried to reprimand him.
- 65. "He introduced the water-tube boilers, which, as every engineer and seaman knows, raise a full head of steam in twenty minutes, instead of twenty hours, as formerly. "He introduced the steam turbine, which was adopted by every nation. "He introduced oil fuel into the Navy, thus making destroyers capable of steaming further, a great benefit being the almost total absence of smoke. He also applied it to battleships and other large craft. "He introduced the Dreadnought, the bulwark of Britain, and the ship that baffled the German nation and made the Kiel Canal useless for years. The oil-burning, water-tubed destroyer, and the Queen Elizabeth—the Secret Service ship and the monitor— all emanated from his brain. "He introduced the battle-cruiser, against the will of a timorous Government whose cry was ever, 'Cut down armaments,' 'Cut down the Army and Navy.' Had Fisher listened, the Germans would to-day have outraged our wives and crucified our children. "He planned the Falkland Islands battle, and sent the Secret Service ships to chase the German submarines out of the Channel. He fought hard against the Dardanelles expedition. "He was Sea Lord when we sank the Blucher, the German destroyers in the North Sea, the German Fleet at the Falklands. "He is a great man, who seems never to have made a mistake." Whilst Sir Edward Grey was giving his explanations in the House of Commons, Lord Devonport was busy in another place. He is one of our shrewdest and most experienced business men. As Chairman of the Port of London Authority and former Parliamentary Secretary of the Board of Trade, he would not be likely to go into figures lightly.
- 66. He had given notice to ask the Government for its official figures of Holland's imports of ore (metal) during 1915. The Duke of Devonshire replied that the figures provided him were only 650,000 tons. It was admitted that Holland had virtually no smelting plant, nor coal to feed it if it had, and the Government was virtually bound to confess that at least this amount of contraband had mostly gone straight through to Germany. Lord Devonport clearly stated that in reality one and a half million tons of metal ore had been imported; whilst he produced statistics showing the name of every ship, the date of entry, the place from which the cargo came, the quantity and character of the ore carried, and the agents to whom each was consigned. To summarise shortly the total shipments for the period named by Lord Devonport, August, 1914, to January 15, 1916, it appears that 298 ships carrying 1,414,311 tons of metal ore entered Rotterdam. The countries from which the ore came included Sweden, Norway, Spain, Algeria, Russia, and Great Britain. The totals shown monthly are as follows: Ore Cargoes. 1914. No. of Ships. Tons. August 38 174,162 September 11 61,679 October 10 47,900 November 8 37,300 December 14 63,900 ——— Total 384,941 1915. January 17 76,200 February 17 79,700 March 13 85,800 April 22 123,800
- 67. May 17 68,100 June 21 95,350 July 21 89,150 August 19 82,300 September 19 92,400 October 22 105,270 November 13 59,700 December 12 48,300 ——— Total 1,006,070 1916. To January 15 4 23,800 ———— Grand Total 1,414,311 Two hundred and fifty eight ships carried 1,321,456 tons of iron ore; 25 ships carried 41,830 tons of zinc ore, the remainder taking copper ore, pyrites, nickel, manganese, and calamine. Lord Devonport added: "What has come of the much-vaunted order in Council declaring that no goods should either enter or leave Germany? What is the ultimate destination of these cargoes? There is no concealment about the matter. Every captain knows exactly. There are no facilities in Holland for converting ore into pig-iron; not a single blast-furnace, and no coal to feed it even if there were. "The cargoes are transhipped into barges and carried up the Rhine to a place in easy communication with Essen, where Krupp's works are situated. Sweden is the main source of the supply. It is astounding to me that the British Government should sit still while these ores are sent to the enemy from mines which are virtually the property of the Swedish Government.
- 68. "Great though the imports of ore into Rotterdam have been, they are insignificant compared with the importations in German ports in the Baltic Sea and the North Sea—Lübeck, Stettin, Swinemunde, Emden and others. From May 1st to December 31st, 1915, the total of those imports were 556 cargoes and 2,089,000 tons of ore. The question is going to become critical for, though the country has been tolerant and long-enduring, things have not gone too well. The sheet-anchor of the situation is the British Fleet." "The figures," says Fairplay, the shipping paper, "sufficiently indicate the absurdity of supposing that the Netherlands Overseas Trust or any similar artificial would-be barrier as at present constituted can, in fact, prevent the enemy from receiving vital supplies of raw or manufactured material." Nineteen days after the delivery of Sir Edward Grey's "blockade" speech in the House of Commons Mr. T. Gibson Bowles, speaking at a great City demonstration in London on February 14th, 1917, under Lord Devonport as Chairman and convened for the purpose of protesting against hampering our Navy, said: "Since the war began Sir Edward Grey had hampered, shackled, and strangled the Fleet in the performance of its duties." Whilst Lord Charles Beresford wrote to the Chairman: "If the Government had used our sea power as they were legally entitled to do at the commencement of the war, by instituting an effective blockade and making all goods entering Germany absolute contraband, the war would now be over." Lord Aberconway added: "The matter is far too serious to be trifled with any longer; my personal knowledge intensifies my conviction." The Government having attempted to evade any direct answer to the startling figures and accusations of the Daily Mail disclosing the get- rich-quick method of the Scandinavian Goulashes, Lord Northcliffe sent a Special Commissioner to Holland, and published the result of his investigations in February, 1916. It showed a repetition of the
- 69. sordid Scandinavian fiasco, a further proof that the so-called blockade was leaking in every seam. To enumerate the masses of statistics would be wearisome. It is sufficient for present purposes to quote a few extracts. Cocoa Beans.—Of the 528 tons imported into Holland in 1916 Germany received the whole. Cocoa Butter.—England could only obtain half what she had in 1913, whereas Germany obtained five times as much. Cocoa Powder.—England obtained half 1913 supplies, whereas Germany obtained approximately ten times as much. Cocoa in Blocks.—In 1913 Germany imported 4 tons from Holland, Belgium none at all; whereas in 1915 no less than 565 tons were exported from Holland into these two countries, all for German use. Copra.—In 1913 Germany obtained 26,728 tons of copra from Holland, whereas in 1915 the amount rose to the amazing total of 106,613 tons. It would appear from the figures that England was indirectly supplying Germany inter alia with margarine. In 1913 Great Britain sent to Holland 1,914 tons of the raw material, as against 6,166 tons in 1916. Germany sent no raw material to Holland during either of the years quoted. In 1913 Holland exported 308 tons of margarine to Belgium and to Germany 401 tons. In 1915 Holland exported 7,616 tons to Belgium and 21,721 tons to Germany. Totals of 709 tons suddenly jumped to 29,237. Coffee.—Before the war Germany had always exported coffee to Holland in thousands of tons. During 1915 she sent in none at all, but she imported from Holland 129,968 tons; whilst 32,822 tons in
- 70. addition were sent to Belgium for German use as against a prior yearly average import of about 8,000 tons. N.B.—England, which during 1911, 1912 and 1918 exported a yearly average of 6,720 tons of coffee to Holland, suddenly increased her exports to this country to 15,672 tons in 1914 and to 28,425 tons in 1915. In March, 1916, Brazil was seizing German ships because she could not collect a trifle of about £4,000,000 owing to her for coffee by the Fatherland. Cotton.—In the three years before the war England exported an average of 7,808 tons of unspun cotton to Holland, but in 1915 she sent no less than 22,856 tons. Germany, which exported an average of 33,975 tons before the war, actually imported from Holland direct in 1915 no less than 38,750 tons. The Commercial Treaty of the Rhine, cunningly made by the clever Teutons before war was declared, prevented the Dutch from even examining any cargoes which were thereunder arranged for direct shipment into Germany; whilst from the very first the workings of the much-boasted arrangement made by our Foreign Office with the Netherlands Overseas Trust piled up evidence, week by week and month by month, that our so-called blockade was an absolute farce. In the famous "Kim" case before the Prize Court, the President, Sir Samuel Evans, made the law quite clear. Figures were placed before the Court to show that the average monthly quantities of lard exported from the United States to all Scandinavia in October and November, 1913, was 427,428 lbs. Within three months of the outbreak of war one company was shipping to Copenhagen alone considerably over twenty times that quantity in three weeks. When it might have been thought that the public had forgotten this complete and overwhelming evidence, Lord Emmott, speaking on behalf of the Government, told the House of Lords that "an abnormal supply to a country is not sufficient reason to stop a cargo." Here was
- 71. a Government spokesman absolutely contradicting the Prize Court Judge—another unwarrantable interference with the rights of Democracy. On February 22nd and 23rd, 1916, the House of Lords debated an important motion ably advocated by Lord Sydenham. "That in conformity with the principle of international law and the legitimate rights of neutrals, more effective use could be made of the Allied Fleets in preventing supplies, directly conducing to the prolongation of the war, from reaching the enemy." Lord Lansdowne, Lord Emmott and the Marquis of Crewe spoke in defence of the Government, but they brought forward no direct proof to upset the alarming statistics which had been quoted against them. Some figures, however, were given to show that during the last past month a greater activity had been caused, in consequence of which there had been some diminution of imports to Germany; whilst it was further promised that as an attempt to concentrate the general supervision of the War Trades Committee the work should be placed in the hands of one Minister, Lord Robert Cecil, who would be given Cabinet rank. That Lord Robert Cecil is a man of great ability no one doubts. The stock he springs from is pedigree so far as politics are concerned, but he is a lawyer. For many years past this country has suffered greatly from a glut of lawyer politicians, particularly in the unwieldy Cabinet of twenty-three members. The nation remembered only too well how this noble lord had fought so strenuously and so persistently against cotton being made contraband. His appointment therefore to this post of vital importance, which could influence, affect and control the duration of the war to such a great extent, was strongly objected to by the public at large. Neither the act nor the man carried an iota of confidence. To have seriously attacked the Government and put it out of office would have raised a general outcry. It was considered disloyal even to criticise. "Wait and see" was the only policy Englishmen were
- 72. permitted to contemplate. Meanwhile this farce, this weakness or this cowardly inaction, whichever epithet is most appropriate to it, was permitted to drift its course. Gleefully the Germans continued to annex the rich cod and herring harvests of Norway, nor did they cavil at the super-price. Gleefully the Norwegian fishermen continued to rake in the deluge of gold, the like of which had never been known within the memory of man. Gleefully the Goulashes of Scandinavia continued to increase and multiply, whilst they prospered and waxed exceedingly rich, in spite of a few widely-proclaimed spectacular fines and confiscations. The advertisements in the papers of neutral countries offering to supply necessities direct into Germany also continued and spread, like the proverbial grain of mustard-seed, until the very mails were glutted with contraband. One of these multitudinous advertisements is given as an example. It is from the Fatherland, March 29th, 1916, the subsidised German- American weekly published in New York: FOOD TO GERMANY. Delivered through my Firm at Stuttgart. Can condensed milk 30 cents Fruit marmalades, per pound35 cents Fifty cigars $2.00 One pound of rice 40 cents One pound of bacon 75 cents One pound of lard 70 cents One pound of cheese 25 cents 100 cigarettes $1.70 Also dried fruits, beans, peas, etc. Invigorating wines for sick and wounded. Information and price lists on request. E. R. Trieler, Dept. F. 35-37, West 23rd St., New York.
- 73. No wonder Lord Grimthorpe, after quoting an influential Frenchman's opinion that "England had muscles of iron but brains of wool," argued that, instead of bringing more lawyers into the management, the country would be much more satisfied if the Ministry of Blockade was put into the hands of a fighting man like Lord Beresford or Lord Fisher. Those in the Secret Service knew that since the outbreak of war Germans had employed only soldiers and sailors to manage it; and that all their lawyers and civilian politicians had been relegated to a back seat until further notice; furthermore, that only proved ability counted. Patronage, length of service, hereditary and social altitude carried no weight whatsoever at Berlin; whilst the capacity for organisation and thoroughness which Germany exhibited had astonished the world. Yea, verily, it is a true saying that "Britishers are the greatest muddlers on earth." It seems to be their grim bulldog pertinacity only which pulls them through, and their individuality which gives them the stamina to stay. As the winter turned to spring and the spring to summer other terrible disasters arose which diverted the attention of the nation from the bogus blockade. Mr. Asquith's "one bright spot," the Mesopotamia expedition, turned to gall and wormwood; the terrible Gallipoli fiasco shocked the nation; the pampered Irish rebels appeared in their true colours; the careless sacrifice of a man whom many believed to be one of the noblest and greatest of Army Chiefs (K. of K.) this world had ever seen, paralysed and numbed every English-speaking land; whilst German spies were still permitted to press their deadly finger-prints upon our national throat owing to our unbelievable weakness in neglecting to intern all aliens of belligerent nationality. Meanwhile the Press continued to growl and to publish statistics from time to time to prove that the so-called blockade was still as great a farce as ever; furthermore, it was absolutely and utterly ineffective to
- 74. stop supplies going to Germany. Whilst Ministers and Members of the Government still had the audacity to refer to its alleged effectiveness and to call attention to the unenviable plight of starving Germany. All true Englishmen should gratefully thank God that we had at least one man amongst the few real men who had the courage of his convictions, namely, Mr. W. M. Hughes, the Australian Premier. He, during his all too short sojourn in the Motherland, rendered noble, great and patriotic service. He called with an unmistakable voice at the British Imperial Council of Commerce in London, on June 8th, 1916, for a real blockade. He said: "Do you realise the tremendous pile of treasure we are pouring out in this contest? Do you think that any nation, no matter how wealthy, can stand indefinitely such a strain on its wealth? It cannot. We are living like spendthrifts, upon our capital. There must come a day when we can no longer live upon it. I want to emphasise the point that we cannot continue this struggle indefinitely. The blockade is one great weapon at our disposal—one of the most effective weapons for shortening the duration of the war—by increasing the pressure upon the enemy. If the blockade had been effective earlier it would have curtailed the war. We now have the power, as Mr. Balfour said, to make that blockade still more effective, and whatever stands in the way of making that blockade effective against the enemy and against neutrals must be swept aside. We have to choose between offending neutrals and inviting defeat. We have to choose between pouring out our treasure and losing the lives of thousands of our best and bravest. Let us hedge around this nation (Germany) a ring of triple steel through which nothing shall pass. I have been told there are still things going out of Britain to Germany. I am told the reason given is that we are getting German money in exchange. That argument does not appeal to me. I would not tolerate the practice for another hour. I would treat those who engage in it as I would treat any other traitor to his country. Therefore insist upon the blockade being such a blockade as will compel our enemies to recognise the power of Britain and the Allies."
- 75. Lord Hugh Cecil, the Blockade Minister, does not appear to have been amongst those present at this memorable gathering. More's the pity of it! Had he been perhaps he might have had his eyes opened at last to the folly and inefficiency of his previous policy and foolishly expressed fallacies. To the probable relief and secret joy of the Cabinet, and to the irreparable loss of the nation, Mr. W. M. Hughes was in the early summer of 1916 compelled to return to his duties in Australia. After his regretted departure the so-called blockade continued to leak, as is proved by the following facts and figures which found their way into the Press in spite of all the hushing-up processes of the weaklings in power. Can it be wondered at that many thousands of astounded Englishmen were actually beginning to believe that some of our prominent Ministers did not want to win the war because they were either indirectly interested financially in Teutonic enterprise, or they were pro-German from other mysteriously concealed causes? What other possible reasons seemed arguable in view of their extraordinary actions, their leaving undone those things which they ought to have done, and their doing those things which they ought not to have done? How German production steadily revived from the shock of the first year of the war is shown by the following table of pig-iron output in tons published in the Berliner Tageblatt: 1914. 1915. 1916. January 1,566,505 874,133 1,077,046 February 1,445,511 803,623 1,033,683 March 1,602,714 938,438 1,114,194 April 1,534,429 938,679 1,073,706 May 1,607,211 985,968 June 1,531,826 993,496 July 1,561,944 1,047,503 August 587,661 1,050,610 September 580,087 1,033,078
- 76. October 734,841 1,076,343 November 788,956 1,019,122 December 858,881 1,029,144 Asking the Prize Court on June 5th, 1916, to condemn the Swedish vessel Hakan, of Gothenburg, with her cargo of 3,238 barrels of salted herrings, the Attorney-General, Sir F. E. Smith, alleged that the fish were intended for Germany. Writing from Lübeck to Gottfried Friedrichs, fishmongers, of Altona, said the Attorney-General, a member of the firm of Witte & Co., their forwarding agents, said: "We have prohibited the export of herrings from Norway, but our firm has obtained a licence to export 50,000 tons. We hope to sell 75,000 tons this winter, so there is plenty of work." Sir Samuel Evans: How many herrings in 50,000 tons? The Attorney-General: My assistants and confederates inform me that there are about 450,000,000 herrings. It is a conservative estimate. These are official figures published by the Netherlands Statistical Department on May 20th, 1916; such great assistance rendered to Germany is more serious owing to the fact that Germany's gain has been our loss. Foodstuffs Sent From Holland, in Tons. (Covering the months January to April.) Eggs— 1914. 1916. To Germany 3,101 11,825 To Britain 2,733 557 Fish— To Germany 21,337 29,378 To Belgium — — Meat— To Germany 4,156 30,621 To Britain 25,460 555 Potato Flour and its products—
- 77. To Germany 13,991 43,861 To Britain 8,831 5,520 Coffee— To Germany 17,429 39,684 Cocoa Powder— To Germany 598 3,302 To Britain 2,155 1,437 Butter— To Germany 4,010 10,237 To Britain 1,387 33 Cheese— To Germany 4,120 25,437 To Britain 5,624 407 One has only to cast the eye down these figures to see what Holland means as a depôt for Germany's food. During the first four months of 1916 Holland had imported by consent of Great Britain 432,702 tons of cereals. No less than 283,792 tons were re-exported from Holland and consequently did not go into home consumption there; 272,630 tons of this went over into Belgium. It is important, also, to note that of the cereals imported 102,722 tons of maize were included in the total. Most of this maize was used for fattening pigs, which were eventually slaughtered and sent to Germany. This abundance of pig food allowed by us to be consumed by the Dutch pigs in fact enabled the Dutch to fatten the immense supply which they sent over to Germany. The meat figures given above must be read in the light of this fact. The more we sent into Holland for her home supply, the more she could release of her home-grown products to the enemy. As between Holland, Germany and ourselves, we lost tremendously. Germany and Holland were of immense assistance to each other, at our expense.
- 78. A weekly circular of the London Rice Brokers' Association shows the following striking contrasts in exports from London: Exports of Rice from London. January 1st to May 27th, 1915. Same period, 1916. Cwt. Cwt. To Holland 247,869 905,078 (say 45,000 tons) To France 22,607 430 Thus the export to Holland had greatly increased and the supply to France had dwindled almost out of existence. During the single week ended May 27th, 1916, 224,252 cwt. (say 11,212 tons) were shipped to Holland from London. On June 2nd, 1916, the London Press wailed over the enormous supplies of grain entering Germany through Roumania, which she was enabled to purchase by exchanging goods made from the raw material permitted so kindly by England to leak through the blockade. In April one consignment of 1,500,000 eggs passed from Holland to Germany in two days only. Indeed, so vast was the drain of Germany upon Holland that the Dutch people complained in June that they were being stinted of their proper food supply. Norway continued to supply nickel, fish, copper, fish oils, and many other things, although England at last awoke in the spring of 1916 to the advisability of purchasing part of the Norwegian fish harvests. In this deal, however, her lawyer Government had not the sense to consult the best export fish merchants, who are essentially business men. She went to work in the usual amateurish way, which spelt reckless waste and extravagance; paying £5 to £7 per package for what could have been previously arranged for at about 10s. or less. The English Government throughout the war had the Norwegian fish trade absolutely in its own hands. Yet one of its own Consuls supplied Germany wholesale in 1914; it supplied coal and salt to assist the
- 79. Germans to garner in practically the entire harvest of 1915; and it was not until the middle of 1916 that some English sluggard in power woke up and paid through the nose for what could have been purchased practically on our own terms. Sweden continued to supply almost everything and anything that Germany required, openly when possible, smuggled in by all manner of tricks and dodges should any difficulty of transport be likely to arise. At the end of June, 1916, a Liverpool merchant contributed some remarkable facts and figures to the Liverpool Courier, proving that England was helping Germany to obtain what she required at the expense of the home consumer in England. The net result of his arguments was that our shipping and home ports were congested for several months by Dutch imports through private arrangements between Holland and England, whereby Holland was supplying Germany to a colossal extent and frustrating the supreme purposes of the so-called blockade. In conclusion, he plaintively besought the nation to adopt the strangle-knot of Mr. Hughes by so tightening the blockade that Holland would no longer be able to provide the Germans with food for her peoples and materials for the manufacture of guns and explosives to slaughter our sons. The tables of figures quoted showed in glaring contrast the usual enormous increases of imports upon pre-war returns which the British reader had grown quite accustomed to see. To give but one example: the shipments of margarine from Holland to Germany during 1915 showed thirteen times greater, etc. On July 20th, 1916, during the hearing of a case in the London Prize Court relating to the S.S. Maracus, the Solicitor-General (Sir George Cave) read an affidavit by Mr. John Hargreaves, provision merchant, Liverpool, stating that in 1915 the price of lard in Germany was 100s. per cwt., as against 50s. in Liverpool. At that price there was an inducement to American shippers to risk shipment to Germany, and to German buyers to open credits in New York. Should the American
- 80. shipper succeed in getting two shipments through, he might well make a large profit which would amply compensate him for the loss of one shipment, apart from his chance of recovering compensation from the British Government. An affidavit by Mr. R. M. Greenwood, Assistant Treasury Solicitor, showed the imports of foodstuffs into Copenhagen during the first six months of 1915 as compared with the similar period of 1913. The figures were: 1913. 1915. Pork 948,400 lbs. 15,062,060 lbs. Lard 3,999,700 lbs. 23,458,720 lbs. Oleo 2,509,900 lbs. 8,775,750 lbs. The evidence in the case proved that the ship was bound for Germany and her captain had been promised a bonus of £200 if the goods reached their destination. On June 28th, 1916, Lord Robert Cecil in reply to a question in the House of Commons, said: "As the result of the Paris Conference His Majesty would be advised to issue an Order in Council withdrawing the successive Orders which had been issued adopting with modifications the Declaration of London, and a general statement should also be issued explaining the reason for this step." Amidst the loud cheering which followed a voice was heard to exclaim, "After twenty-three months!" How Potsdam must have hugged itself with delight in 1909, 1910, and 1911 at the absurdly childish simplicity exhibited by the English Liberal Government in nullifying all its geographical advantages by accepting such a one-sided code of sea-law which gave Germany the right to stop food en route to British ports, while forbidding Great Britain to stop food en route to Germany, and whilst in force rendered any effective blockade of Germany impossible.
- 81. But what powerful mysterious motives prompted its re-adoption after it had been rejected by the House of Lords? Again on August 20th, 1914, why did the Cabinet illegally put it into force with modifications —though Article 65 thereof states that the code is indivisible? What was held in the unseen hand and to whom was it extended? On August 2nd, 1916, M. Clemenceau published an article in L'Homme Enchainé, headed, "A Fresh Assassination," in which, after commenting upon the brutal murders of Nurse Cavell and the innocent Captain Fryatt, he wrote: "It is time that Great Britain made the weight of her will felt, especially as regards the strict application of the blockade, which, has too often been relaxed out of a desire not to arouse an unpleasant quarrel with Washington. It is time to end these half-measures. We must make up our minds as to what to do, and do it." On July 6th, 1916, Lord Robert Cecil admitted in the House of Commons, in reply, what was tantamount to a confession that the British Fleet employed in the blockade was still muzzled, being bound down by red-tape precedents and strict London directions. On July 9th he was further compelled to confess that 10,708 tons of lard had been permitted to enter Belgium, as well as about 2,000 tons of tallow and other fats. Nominally this was fathered by the Neutral Relief Committee, but in reality it was just so much more assistance granted to the enemy. Fat (for Explosives) in tons In the early part of 1914 Germany exported lard to Holland, but this ceased on the eve of war. Great Britain, on the other hand, for some extraordinary and unintelligible reason, permitted her exports to Holland to increase. These are the figures: From Germany. From Great Britain. 1914 861 660 1915 Nil 6,591
- 82. 1916 Nil 12,273 Barley for Malt In 1916 Great Britain exported to Holland about fifteen times more barley than normal pre-war exports, so diminishing our home supplies that the British working-man was deprived of his national beverage through shortage and prohibitive prices. Whisky also was similarly affected. Tobacco The Christian spirit of "love your neighbours and your enemies better than yourselves" had apparently no limits with the British Government. Their loyal and hard-suffering subjects were deprived of a full supply of the soothing weed on the excuse of economising freight room, but no effort seems to have been made to curtail Dutch supplies, which were thirty-five times greater than the pre-war exports. In 1914 Hamburg and Bremen exported 4,544 tons of tobacco to Holland, but in 1915 and 1916 neither of these towns exported any at all. The amounts exported by Holland from January to June in tons were as follows: To Great Britain. To Germany. 1914 1,611 31,891 1915 1,672 54,456 1916 923 96,931 The figures published by the German Steel and Iron Manufacturers Association for the first six months of each respective year show the following outputs, thanks to Sir Francis Oppenheimer's previous Netherlands Overseas Trust, which permits iron ore in millions of tons to proceed direct to Krupps' and other blast furnaces in Germany without let or hindrance to be used against us.
- 83. Pig Iron Tons 1915 5,530,000 1916 6,497,000 Steel 1915 6,187,000 1916 7,756,000 The Lokal Anzeiger, July 28th, 1916, remarked: "These figures constitute a most gratifying state of affairs in respect of the requirements of the German Armies." No wonder the captured German officer remarked: "You English will always be fools, whilst we Germans can never be gentlemen"! In August[25] a Mr. E. Bell, of 12, Yarborough Road, Lincoln, wrote to the Press as follows: "The talk of tightening the blockade of Germany is rather futile in face of the following Board of Trade figures referring to cotton yarn exported from the United Kingdom to the following neutral countries: June Sweden Norway Denmark Holland Switzerland 1914 108,900 218,700 106,400 3,220,800 722,600 1915 260,800 348,300 204,700 4,493,300 1,788,800 1916 279,200 508,200 598,400 7,539,800 1,304,100 "Germany is obviously getting the surplus." The values[26] of New York exports taken for the week July 30th to August 5th are equally startling: 1915. 1916. New York to £ £ Norway 1,884 137,176
- 84. Holland 713 717,601 Holland and Scandinavia 123,327 970,255 On August 26th, 1916, an agreement was signed between the Dutch Fishing Association and the British Government regarding the release of some 120 to 150 Dutch fishing-boats laid up in Scottish ports, whereby not more than 20 per cent. of their catch shall be permitted to go to Germany. Of the remainder twenty per cent. was to be retained for home consumption, and sixty per cent. sold to neutral countries. On each barrel of this sixty per cent. the good, kind, benevolent British Government agreed to pay a subsidy of 30s. to the Dutch boat-owners. Now the D.F.A. owned about 850 vessels and 1,000 barrels is a good average season's catch! In addition to this arrangement the British Government agreed to pay full compensation for their loss of part of the season, to be calculated on the basis of the returns on an average season. They also agreed to pay for any damage which might have happened to the interned boats.[27] One wonders what British fishermen whose vessels have been commandeered had to say when they were informed of these facts. The Hamburger Nachrichten of August 23rd, 1916, published a telegram from its Hague correspondent declaring that the semi- official German Central Purchase Company was seizing Dutch food in enormous quantities; that local merchants were in a state of alarm and threatening Government interference; and their correspondent defiantly stated: "The Netherlands Government will hardly dream of interfering with the activity of the Dutch Bureau of the German Central Purchase Company, the operations of which are assuming larger and larger dimensions." To add further proof of the utter futility and hollow sham of the alleged blockade safeguards, namely, the Danish Association Agreement and the Netherlands Overseas Trust, Sir Henry Dalziel
- 85. informed the House of Commons on August 22nd, 1916, that in June Denmark imported over ten times as much cotton yarn as in June, 1913, and that in the first six months of the present year Holland exported to Germany over twenty times as much butter as in the first six months of 1914, nearly eight times as much cheese, and over seven times as much meat. The unfortunate Lord Robert Cecil in mid-August gave quite a eulogistic report upon his stewardship as Blockade Minister, which was immediately followed by the arrival from New York of the Custom House returns showing that during the week ending August 5th the value of the exports to Holland, Norway, Sweden, and Denmark was eight times as great as in the corresponding week of the preceding year. To Holland the exports had increased in value a thousandfold and to Norway seventy-fivefold. On September 1st, 1916, the Government, through the War Trade Statistical Department, issued to the Press an official Memorandum on the question of the efficacy of the British blockade. It barely amounted to the proverbial half-truth, and was pitiably feeble. It was more than unfortunate that the Government should rush into print just before the United States export figures were due for publication—only a week later. These latter reliable statistics showed an extraordinary state of affairs: Exports from U.S.A. 1914. 1915. 1916. £ £ £ To Norway 1,813,400 7,815,000 10,735,600 To Sweden 2,928,800 15,654,800 10,387,800 To Denmark 3,134,000 15,964,800 11,132,400 To Holland 22,443,200 28,653,400 19,852,600 To Switzerland 204,000 547,200 1,631,200
- 86. The Telegraaf, Amsterdam's leading journal, on September 11th, 1916, quoted Governmental statistics to account for the excessive rise in price of her home products, concluding by the statement that "Holland has sold her livelihood for greater war profits"; whilst all the Dutch Press seemed to deplore mildly the vast and unmanageable manner in which the smuggling of goods over the German frontier was permitted to continue. The figures for meat, cheese, eggs, vegetables, and butter showed an average increased export of seventy-five per cent. on preceding years. Practically every ounce went to Germany or to territory under her rule. On September 12th, 1916, Reuter's representative at the Hague was able to announce that: "The Dutch Overseas Trust had obtained the release of 420 tons of Kapok, Java cotton, and had also succeeded in removing the difficulties in the way of the importation of cocoa- beans." Such paragraphs as the above could be found repeatedly by anyone who chose to search the Press. No wonder the smouldering wrath of the long-suffering British public became fanned to a flame and its confidence in its so-called representative Ministers correspondingly decreased. On September 9th, 1916, the Foreign Office issued a notice that no further export licenses or further facilities would be given by H.M.G. for the importation of certain specified commodities until further notice. The list embraced scores of foods, but, in fact, was merely another patch to the very ragged mantle covering the so-called blockade. On September 12th, 1916, the War Trade Statistical Department made another feeble attempt in public to refute the statistics quoted by the Press. It set out specious and plausible arguments why general conclusions should be drawn in a light more favourable to our interests. It gave no denials nor suggested that the figures quoted
- 87. were not correct. It was a fretful official apology, a tacit admission of weakness and inefficiency. A casual remark was made by a really able German in the Wilhelmstrasse on English policy in regard to Germany, to Mr. D. T. Curtin, as reported by him in the Times, October 21st, 1916. "He said to me: "'When the war began we thought it would be a fight between the German Army and the British Navy. That was the cause of the outbreak of German anger against England on August 4th, 1914. As time went on we found that the English Government drew the teeth of its Navy and enabled us to get in through the then so-called blockade supplies of cotton, copper, lubricating oil, wool' (here he named some twenty commodities) 'in a sufficiency that will last us many long months yet. How different would have been our position to-day if the British Navy had controlled the blockade as we had every reason to fear it would! We can and will hold out for a long time, thanks to their blunders.' "Blockade policy, prisoner policy, enemy trade control, the Zeppelin reprisal policy—all these are puzzles to the rulers of Germany. All are taken as part and parcel of their belief of your desire to curry favour with them and your fear of their after-the- war trade struggle. "The average German holds similar views as to America's fear of the Kaiser's Army and Navy after the war. They frankly tell us that it will be our turn next." On October 25th, 1916, Mr. D. T. Curtin explained in the Times how, when he was in Germany, a neutral and pro-Ally resident of a certain port in Germany with whom he discussed things took him for a walk and showed him the quays. There were not hundreds, but thousands of barrels of fats. "It almost makes me weep," he said, "to know that every one of these barrels lengthens the war and destroys the lives of
- 88. gallant soldiers and their officers." And apart from the public evasions of the blockade is the secret smuggling—difficult to deal with. A day or so previously Mr. Curtin had written: "Every bar of chocolate entering Germany prolongs the war, which I know from my own personal necessities. The Allies and the Government should realise the great value of the utmost pressure of the blockade." It was not until December, 1916, that the rising tide of public feeling threatened to burst the banks of reasonable control. On the first day of that month a crowded meeting of City business men was held in the Cannon Street Hotel under the presidency of Lord Leith of Fyvie to protest against the slackness of the Government and terrible blunders which were far too serious to openly discuss; in particular to insist that "the British Navy be set free to exercise to the full all its lawful sea powers." Startling disclosures were made, and the Government, which had twice restored itself after its legal expiration, was characterised as worn-out and stale, unable to make peace any more than it was able to make war; sentiments which were unanimously acclaimed. Almost the entire British Press echoed this condemnation, and the Haldane group, recognising that discretion was the better part, awoke at last from its delusions of the value placed by the nation upon their personal services, and after a few feeble remonstrances retired in favour of a new Cabinet. "Wait and see" was compelled to give place to "Do it now." Mr. Asquith the Unready, Lord Grey of Falloden, the Irresolute, Lord Haldane, the friend of the Kaiser, and the Simonite group of backers, who for fifteen unlucky years had so grievously and disastrously led the country astray; who had cut down armaments, hoodwinked the nation, and when war was declared held back conscription, muzzled the Fleet and were too late for everything, were at last fallen from doing further mischief, and the nation breathed its prayers of thankfulness.
- 89. Of the late Prime Minister (Mr. Asquith) one able editor wrote: "Never before in all our history have such opportunities been given. He had no opposition; the nation was solid; the Empire was behind him. No country has ever given any leader such devotion and none has ever seen its devotion so carelessly wasted. Declaring he would 'stick at nothing,' he stuck at everything, and moved only when he was pushed."[28] What Germany thought of the change is reflected in an extract from its Press when it first heard of the resignation of Mr. David Lloyd George from the War Office, and it was under the belief that the Haldane group had triumphed over him. The Bavarian Courier, December 5th, 1916, said: "This is a terrible disaster for the war party in England," whilst the Leipzig Tageblatt said: "The British people have doubtless had enough of this war agitator. His fall from power brings nearer an honourable peace for Germany." Within a few days of Mr. Lloyd George being created Prime Minister of England the Kaiser was seeking peace. Res ipsa loquitur. * * * * * * What has been given is merely a rough and very deficient resumé of England's sham blockade, which was permitted to muddle along its costly, tragic, and fatal course until the Americans joined the Allies in their fight for freedom and the rights of small nations. Washington at once swept aside maudlin sentiment by its practical common sense, get-right-there-quick decisions. The nation's relief cannot be expressed in words. Was it to be wondered at that from the soul of the Motherland prayers had so long and so often ascended? "Oh, for a man of the old, old Viking blood to lead and direct the battle in place of those poor craven lawyer politicians in the Cabinet
- 90. Welcome to our website – the perfect destination for book lovers and knowledge seekers. We believe that every book holds a new world, offering opportunities for learning, discovery, and personal growth. That’s why we are dedicated to bringing you a diverse collection of books, ranging from classic literature and specialized publications to self-development guides and children's books. More than just a book-buying platform, we strive to be a bridge connecting you with timeless cultural and intellectual values. With an elegant, user-friendly interface and a smart search system, you can quickly find the books that best suit your interests. Additionally, our special promotions and home delivery services help you save time and fully enjoy the joy of reading. Join us on a journey of knowledge exploration, passion nurturing, and personal growth every day! ebookbell.com