ARROW-5: Update drill-fmpp-maven-plugin to 1.5.0 #1

lw-lin · 2016-02-19T15:24:59Z

Currently when we run maven install, we'll get:

org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.apache.drill.tools:drill-fmpp-maven-plugin:1.4.0:generate (generate-fmpp) on project vector: Execution generate-fmpp of goal org.apache.drill.tools:drill-fmpp-maven-plugin:1.4.0:generate failed: Plugin org.apache.drill.tools:drill-fmpp-maven-plugin:1.4.0 or one of its dependencies could not be resolved: Failure to find org.freemarker:freemarker:jar:2.3.24-SNAPSHOT in http://repository.apache.org/snapshots was cached in the local repository, resolution will not be reattempted until the update interval of apache.snapshots has elapsed or updates are forced

Reason is: the freemaker.version that takes effect is the one defined by drill-fmpp-maven-plugin 's dependency drill-root(which is 2.3.24-SNAPSHOT), rather than the one defined by arrow-java-root(which is 2.3.21).

This PR adds explicit dependencies on fmpp and frermaker for drill-fmpp-maven-plugin, making the fmpp.version and freemaker.version defined by arrow-java-root to take effect.

lw-lin · 2016-02-19T15:30:28Z

hi @jacques-n @wesm , would you mind taking a look at this when you have time? Thanks.

jacques-n · 2016-02-19T15:45:08Z

It looks like drill-fmpp-maven-plugin 1.5 is available now. Can you look and see if updating to that plugin fixes the issue?

https://repository.apache.org/#nexus-search;gav~~drill-fmpp-maven-plugin~1.5.0~~

lw-lin · 2016-02-20T01:07:43Z

Sure, thanks for pointing this out, @jacques-n.

Updating to 1.5.0 does make mvn install complete without errors, since freemaker.verion is now specified as 2.3.1 in drill-root-1.5.0.pom.

However, fmpp.version/freemaker.version specified in arrow-java-root still don't take effect by this simple updating. Seems like we should either make changes in this PR's first commit to make fmpp.version/freemaker.version take effect, or drop the fmpp.version/freemaker.version in arrow-java-root because it would be some kind of misleading.

Thoughts please?

jacques-n · 2016-02-20T01:28:20Z

Let's just remove the fmpp/freemarker version numbers.

lw-lin · 2016-02-20T01:57:39Z

Sure. Issued two more commits, and updated this PR's summary as well. Mind taking another look please?

jacques-n · 2016-02-20T02:40:35Z

Looks good. Will merge shortly. Thanks for the PR.

lw-lin · 2016-02-20T02:43:37Z

Thanks for reviewing this!

AllenFang · 2016-02-20T04:34:48Z

Thanks for fixing 👍

…on format. Author: Wes McKinney <[email protected]> Author: Nong Li <[email protected]> Closes #292 from nongli/file and squashes the following commits: 18890a9 [Wes McKinney] Message fixes. Fix Java test suite. Integration tests pass f187539 [Nong Li] Merge pull request #1 from wesm/file-change-cpp-impl e3af434 [Wes McKinney] Remove unused variable 664d5be [Wes McKinney] Fixes, stream tests pass again ba8db91 [Wes McKinney] Redo MessageSerializer with unions. Still has bugs 21854cc [Wes McKinney] Restore Block.bodyLength to long 7c6f7ef [Nong Li] Update to restore Block behavior 27b3909 [Nong Li] [ARROW-499]: [Java] Update file serialization to use the streaming serialization format.

…XXX in plasma protocol. Related to #878, add DCHECK for ReadXXX. Author: Yeolar <[email protected]> Closes #887 from Yeolar/fixtypo_plasma_and_add_DCHECK and squashes the following commits: 4df63bc [Yeolar] clang-format for too long lines. 143d254 [Yeolar] Update, compile passed. 09ff103 [Yeolar] Fix conflicts. b951d8d [Yeolar] Merge pull request #1 from apache/master ebae611 [Yeolar] Fix typo in plasma protocol; add DCHECK for ReadXXX in plasma protocol.

…ties As per #872 I am upgrading Jackson to the latest version on the current train (2.7.1 --> 2.7.9) Author: Matt Darwin <(none)> Author: Matt <[email protected]> Closes #929 from mattdarwin/ARROW-1242-upgrade-jackson and squashes the following commits: d059517 [Matt Darwin] 1242 upgraing jackson to 2.7.9 bc3b6a0 [Matt] Merge pull request #1 from apache/master

NB this commit excludes Jackson and logback upgrades, since they are dealt with in 871 and 872 Author: Matt Darwin <(none)> Author: Matt Darwin <[email protected]> Author: Matt <[email protected]> Closes #873 from mattdarwin/upgrade-libs and squashes the following commits: 9b51f46 [Matt Darwin] Merge branch 'master' into upgrade-libs 284a4ce [Matt Darwin] Merge branch 'master' of https://github.com/apache/arrow 79550b1 [Matt Darwin] rolling back lilith to 0.9.44 since 8 doesn't support java 7 c63eef6 [Matt Darwin] Merge branch 'master' into upgrade-libs bc3b6a0 [Matt] Merge pull request #1 from apache/master 8599ba0 [Matt Darwin] backing out guava upgrade 80d81e6 [Matt Darwin] downgrading guava to 20 for java 7 compatibility 806f348 [Matt Darwin] Merge branch 'master' into upgrade-libs 8aafb7e [Matt Darwin] correcting indentation in BaseValueVector 94c1469 [Matt Darwin] upgrading netty to 4.0.49 cff5596 [Matt Darwin] reverting to netty 4.0.41.Final 568737d [Matt Darwin] switching to Collections from Guava for empty iterator c194e48 [Matt Darwin] upgraded hppc to 0.7.2 38be468 [Matt Darwin] upgrading libs except jackson and logback

…(take 2) sorry, this was still not fixed properly. logback version is separately specified in 2 places. Fixed properly this time. Author: Matt Darwin <(none)> Author: Matt <[email protected]> Closes #960 from mattdarwin/ARROW-1240-upgrade-logback and squashes the following commits: 3492f66 [Matt Darwin] upgrading logback in tools/pom.xml 206b48d [Matt Darwin] Merge branch 'master' into ARROW-1240-upgrade-logback 284a4ce [Matt Darwin] Merge branch 'master' of https://github.com/apache/arrow bc3b6a0 [Matt] Merge pull request #1 from apache/master 3e2f676 [Matt Darwin] Merge branch 'master' into ARROW-1240-upgrade-logback caed163 [Matt Darwin] upgrading slf4j to 1.7.25

…ties (take 2) sorry, PR #929 failed to actually change the Jackson version, since the `jackson.version` variable defined in java/pom.xml is not used in java/vector/pom.xml That's now fixed in this PR. Author: Matt Darwin <(none)> Author: Matt <[email protected]> Closes #957 from mattdarwin/ARROW-1242-upgrade-jackson and squashes the following commits: ad15e5f [Matt Darwin] Merge branch 'master' into ARROW-1242-upgrade-jackson ee29d65 [Matt Darwin] Merge branch 'master' of https://github.com/apache/arrow into ARROW-1242-upgrade-jackson 06d7745 [Matt Darwin] upgrading jackson to 2.7.9 PROPERLY this time... 284a4ce [Matt Darwin] Merge branch 'master' of https://github.com/apache/arrow d059517 [Matt Darwin] 1242 upgraing jackson to 2.7.9 bc3b6a0 [Matt] Merge pull request #1 from apache/master

…is alive before enqueue new record when download file. use pyarrow download file will raise queue.Full exceptions sometimes. jira: https://issues.apache.org/jira/browse/ARROW-2002 Author: kmiku7 <[email protected]> Closes #1485 from kmiku7/master and squashes the following commits: 8d5f905 [kmiku7] fix queue.FULL exception when writer thread write data slowly. 722182b [kmiku7] Merge pull request #1 from apache/master

…lue data Modified BinaryBuilder::Resize(int64_t) so that when building BinaryArrays with a known size, space is also reserved for value_data_builder_ to prevent internal reallocation. Author: Panchen Xue <[email protected]> Closes #1481 from xuepanchen/master and squashes the following commits: 707b67b [Panchen Xue] ARROW-1712: [C++] Fix lint errors 360e601 [Panchen Xue] Merge branch 'master' of https://github.com/xuepanchen/arrow d4bbd15 [Panchen Xue] ARROW-1712: [C++] Modify test case for BinaryBuilder::ReserveData() and change arguments for offsets_builder_.Resize() 77f8f3c [Panchen Xue] Merge pull request #5 from apache/master bc5db7d [Panchen Xue] ARROW-1712: [C++] Remove unneeded data member in BinaryBuilder and modify test case 5a5b70e [Panchen Xue] Merge pull request #4 from apache/master 8e4c892 [Panchen Xue] Merge pull request #3 from xuepanchen/xuepanchen-arrow-1712 d3c8202 [Panchen Xue] ARROW-1945: [C++] Fix a small typo 0b07895 [Panchen Xue] ARROW-1945: [C++] Add data_capacity_ to track capacity of value data 18f90fb [Panchen Xue] ARROW-1945: [C++] Add data_capacity_ to track capacity of value data bbc6527 [Panchen Xue] ARROW-1945: [C++] Update test case for BinaryBuild data value space reservation 15e045c [Panchen Xue] Add test case for array-test.cc 5a5593e [Panchen Xue] Update again ReserveData(int64_t) method for BinaryBuilder 9b5e805 [Panchen Xue] Update ReserveData(int64_t) method signature for BinaryBuilder 8dd5eaa [Panchen Xue] Update builder.cc b002e0b [Panchen Xue] Remove override keyword from ReserveData(int64_t) method for BinaryBuilder de318f4 [Panchen Xue] Implement ReserveData(int64_t) method for BinaryBuilder e0434e6 [Panchen Xue] Add ReserveData(int64_t) and value_data_capacity() for methods for BinaryBuilder 5ebfb32 [Panchen Xue] Add capacity() method for TypedBufferBuilder 5b73c1c [Panchen Xue] Update again BinaryBuilder::Resize(int64_t capacity) in builder.cc d021c54 [Panchen Xue] Merge pull request #2 from xuepanchen/xuepanchen-arrow-1712 232024e [Panchen Xue] Update BinaryBuilder::Resize(int64_t capacity) in builder.cc c2f8dc4 [Panchen Xue] Merge pull request #1 from apache/master

@trxcllnt

This PR moves the `Table` class out of the Vector hierarchy and adds optimized dataframe operations to it. Currently implements an optimized `scan()` method, `filter(predicate)`, `count()`, and `countBy(column_name)` (only works on dictionary-encoded columns). Some usage examples, based on the file generated by `js/test/data/tables/generate.py`: ``` js > let table = Table.from(...); > table.count() 1000000 > table.filter(col('lat').gteq(0)).count() 499718 > table.countBy('origin').toJSON() { Charlottesville: 166839, 'New York': 166251, 'San Francisco': 166642, Seattle: 166659, 'Terre Haute': 166756, 'Washington, DC': 166853 } > table.filter(col('lng').gteq(0)).countBy('origin').toJSON() { Charlottesville: 83109, 'New York': 83221, 'San Francisco': 83515, Seattle: 83362, 'Terre Haute': 83314, 'Washington, DC': 83479 } ``` There are performance tests for the dataframe operations, to run them you must first generate the test data by running `npm run create:perfdata`. The PR also includes @trxcllnt's refactor of the JS implementation to make it more closely resemble the C++ implementation. This refactor resolves multiple JIRAs: ARROW-1903, ARROW-1898, ARROW-1502, ARROW-1952 (partially), and ARROW-1985 Author: Paul Taylor <[email protected]> Author: Brian Hulette <[email protected]> Author: Brian Hulette <[email protected]> Closes #1482 from TheNeuralBit/table-scan-perf and squashes the following commits: 52f1e0e [Brian Hulette] <, > are not commutative, misc cleanup 04b1838 [Brian Hulette] even more table tests 16b9ccb [Brian Hulette] Merge pull request #4 from trxcllnt/js-cpp-refactor fe300df [Paul Taylor] fix closure es5/umd toString() iterator 3d5240a [Paul Taylor] fix more externs 10c48ad [Paul Taylor] Merge branch 'table-scan-perf' of github.com:ccri/arrow into js-cpp-refactor dbe7f81 [Brian Hulette] Add more Table unit tests 1910962 [Brian Hulette] Add optional bind callback to scan 5bdf17f [Brian Hulette] Fix perf 8cf2473 [Brian Hulette] Merge remote-tracking branch 'origin/master' into table-scan-perf 4a41b18 [Paul Taylor] add src/predicate to the list of exports we should save from uglify 5a91fab [Paul Taylor] add more view, predicate externs f6adfb3 [Brian Hulette] Create predicate namespace f7bb0ed [Paul Taylor] Merge branch 'table-scan-perf' of github.com:ccri/arrow into js-cpp-refactor e148ee4 [Paul Taylor] Merge branch 'extern-woes' into js-cpp-refactor 25cdc4a [Paul Taylor] add src/predicate to the list of exports we should save from uglify dc7c728 [Paul Taylor] add more view, predicate externs 25e6af7 [Brian Hulette] Create predicate namespace 579ab1f [Brian Hulette] Merge pull request #2 from trxcllnt/js-cpp-refactor f3cde1a [Paul Taylor] fix lint 9769773 [Paul Taylor] fix vector perf tests 016ba78 [Brian Hulette] Merge pull request #1 from trxcllnt/js-cpp-refactor 272d293 [Paul Taylor] Merge pull request #4 from ccri/empty-table 7bc7363 [Brian Hulette] Fix exception for empty Table 8ddce0a [Paul Taylor] check bounds in getChildAt(i) to avoid NPEs f1dead0 [Paul Taylor] compute chunked nested childData list correctly 18807c6 [Paul Taylor] rename ChunkData's fields so it's more clear they're not semantically similar to other similarly named fields 7e43b78 [Paul Taylor] add test:integration npm script a5f200f [Paul Taylor] Merge pull request #3 from ccri/table-from-struct c8cd286 [Brian Hulette] Add Table.fromStruct a00415e [Brian Hulette] Fix perf 54d4f5b [Paul Taylor] lazily allocate table and recordbatch columns, support NestedView's getChildAt(i) method in ChunkedView 40b3638 [Paul Taylor] run integration tests with local data for coverage stats fe31ee0 [Paul Taylor] slice the flat data values before returning an iterator of them e537789 [Paul Taylor] make it easier to run all integration tests from local data c0fd2f9 [Paul Taylor] use the dictionary of the last chunked vector list for chunked dictionary vectors e33c068 [Paul Taylor] Merge pull request #2 from ccri/fixed-size-list 5bb63af [Brian Hulette] Don't read OFFSET vector for FixedSizeList 614b688 [Paul Taylor] add asEpochMs to date and timestamp vectors 87334a5 [Paul Taylor] Merge branch 'table-scan-perf' of github.com:ccri/arrow into js-cpp-refactor b7f5bfb [Paul Taylor] rename numRows to length, add table.getColumn() e81082f [Paul Taylor] export vector views, allow cloning data as another type 700a47c [Paul Taylor] export visitors e859e13 [Paul Taylor] fix package.json bin entry 0620cfd [Brian Hulette] use Math.fround 0126dc4 [Brian Hulette] Don't recompute total length e761eee [Brian Hulette] Rename asJSON to toJSON 6c91ed4 [Paul Taylor] Merge branch 'master' of github.com:apache/arrow into js-cpp-refactor-merge_with-table-scan-perf d2b18d5 [Paul Taylor] Merge remote-tracking branch 'ccri/table-scan-perf' into js-cpp-refactor-merge_with-table-scan-perf f3f3b86 [Paul Taylor] rename table.ts to recordbatch.ts in preparation for merging latest e3f629d [Paul Taylor] fix rest of the mangling issues fa7c17a [Paul Taylor] passing all tests except es5 umd mangler ones e20decd [Brian Hulette] Add license headers edcbdbe [Brian Hulette] cleanup 20717d5 [Brian Hulette] Fixed countBy(string) 7244887 [Brian Hulette] Add table unit tests... 6719147 [Brian Hulette] Add DataFrame.countBy operation 2f4a349 [Brian Hulette] Minor tweaks 2e118ab [Brian Hulette] linter a788db3 [Brian Hulette] Cleanup a9fff89 [Brian Hulette] Move Table out of the Vector hierarchy 1d60aa1 [Brian Hulette] Moved DataFrame ops to Table. DataFrame is now an interface e8979ba [Brian Hulette] Refactor DataFrame to extend Vector<StructRow> 6a41d68 [Brian Hulette] clean up table benchmarks 2744c63 [Brian Hulette] Remove Chunked/Simple DataFrame distinction aa999f8 [Brian Hulette] Add DictionaryVector optimization for equals predicate 4d9e8c0 [Brian Hulette] Add concept of predicates for filtering dataframes 796f45d [Brian Hulette] add DataFrame filter and count ops 30f0330 [Brian Hulette] Add basic DataFrame impl ... a1edac2 [Brian Hulette] Add perf tests for table scans d18d915 [Paul Taylor] fix struct and map rows 61dc699 [Paul Taylor] WIP -- refactor types to closer match arrow-cpp 62db338 [Paul Taylor] update dependencies and add es6+ umd targets to jest transform ignore patterns to fix ci 6ff18e9 [Paul Taylor] ship es2015 commonJS in main package to avoid confusion 74e828a [Paul Taylor] fix typings issues (ARROW-1903)

Bootstrap evaluation using llvm code generation LLVM code generation is done using a mix of : - glue IR code that loops over the vector, generates function calls (and) - byte-code files generated from simple c++ functions using clang (emit-llvm). The glue-code and pre-compiled byte code are merged and optimized together. Expressions are specified using a "tree builder" where each node is an arrow vector, or a binary/unary function. During code generation, the expressions are "decomposed" so that the value array and bitmap array are evaluated separately to compute the expression result. This avoids the use of too many branch/conditional instructions (checks for "if null"), and hence, can be vectorized efficiently. Support added for arithmetic and logical expressions on numeric types. Travis CI support added for build on ubuntu.

PR comments from apache#34311

We should set `valueCount` when loading `NullVector`. Unfortunately there's no `nullCount` field.

Minor fixes to repair AVX2 build

This is ugly, but necessary because assuming CleanOffsets always produces a Buffer that can be sliced by offsets.offset() is wrong.

…ormatting buffer With ASAN, this reproduces the issue. [ RUN ] Formatting.Timestamp ================================================================= ==4191383==ERROR: AddressSanitizer: stack-buffer-overflow on address 0x7fff1804c48f at pc 0x5608edffe39d bp 0x7fff1804c110 sp 0x7fff1804c108 WRITE of size 1 at 0x7fff1804c48f thread T0 #0 0x5608edffe39c in arrow::internal::detail::FormatOneChar(char, char**) /home/felipeo/code/arrow/cpp/src/arrow/util/formatting.h:132:67 apache#1 0x5608ee035c00 in arrow::internal::detail::FormatYYYY_MM_DD(arrow_vendored::date::year_month_day, char**) /home/felipeo/code/arrow/cpp/src/arrow/util/formatting.h:351:5 apache#2 0x5608ee05e8a0 in decltype(std::declval<arrow::StringAppender&>()(std::basic_string_view<char, std::char_traits<char> >{})) arrow::internal::StringFormatter<arrow::TimestampType, void>::operator()<std::chrono::duration<long, std::ratio<1l, 1000l> >, arrow::StringAppender&>(std::chrono::duration<long, std::ratio<1l, 1000l> >, long, arrow::StringAppender&) /home/felipeo/code/arrow/cpp/src/arrow/util/formatting.h :521:5 apache#3 0x5608ee05d60f in decltype(std::declval<arrow::internal::StringFormatter<arrow::TimestampType, void>&>()(std::chrono::duration<long, std::ratio<1l, 1l> >{}, std::declval<long&>(), std::declval<arrow::StringAppender&>())) arrow::util::VisitDuration<arrow::internal::StringFormatter<arrow::TimestampType, void>&, long&, arrow::StringAppender&>(arrow::TimeUnit::type, arrow::internal::StringFormatter<arrow::Timestam pType, void>&, long&, arrow::StringAppender&) /home/felipeo/code/arrow/cpp/src/arrow/util/time.h:60:14 apache#4 0x5608ee05d122 in decltype(std::declval<arrow::StringAppender&>()(std::basic_string_view<char, std::char_traits<char> >{})) arrow::internal::StringFormatter<arrow::TimestampType, void>::operator()<arrow::StringAppender&>(long, arrow::StringAppender&) /home/felipeo/code/arrow/cpp/src/arrow/util/formatting.h:527:12 apache#5 0x5608edfeffb3 in void arrow::AssertFormatting<arrow::internal::StringFormatter<arrow::TimestampType, void>, long>(arrow::internal::StringFormatter<arrow::TimestampType, void>&, long, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) /home/felipeo/code/arrow/cpp/src/arrow/util/formatting_util_test.cc:52:3 apache#6 0x5608edfece95 in arrow::Formatting_Timestamp_Test::TestBody() /home/felipeo/code/arrow/cpp/src/arrow/util/formatting_util_test.cc:540:5 apache#7 0x7fd95d7901de in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (/home/felipeo/code/arrow/cpp/ninja/debug/libarrow_testing.so.1600+0xd901de) (BuildId: dd9af0bafdb1786050262e8f6002568a9f08ecf6) apache#8 0x7fd95d784905 in testing::Test::Run() (/home/felipeo/code/arrow/cpp/ninja/debug/libarrow_testing.so.1600+0xd84905) (BuildId: dd9af0bafdb1786050262e8f6002568a9f08ecf6) apache#9 0x7fd95d784a84 in testing::TestInfo::Run() (/home/felipeo/code/arrow/cpp/ninja/debug/libarrow_testing.so.1600+0xd84a84) (BuildId: dd9af0bafdb1786050262e8f6002568a9f08ecf6) apache#10 0x7fd95d785038 in testing::TestSuite::Run() (/home/felipeo/code/arrow/cpp/ninja/debug/libarrow_testing.so.1600+0xd85038) (BuildId: dd9af0bafdb1786050262e8f6002568a9f08ecf6) apache#11 0x7fd95d78573e in testing::internal::UnitTestImpl::RunAllTests() (/home/felipeo/code/arrow/cpp/ninja/debug/libarrow_testing.so.1600+0xd8573e) (BuildId: dd9af0bafdb1786050262e8f6002568a9f08ecf6) apache#12 0x7fd95d7907a6 in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) (/home/felipeo/code/arrow/cpp/ninja/debug/libarrow_testing.so.1600+0xd907a6) (BuildId: dd9af0bafdb1786050262e8f6002568a9f08ecf6) apache#13 0x7fd95d784b4b in testing::UnitTest::Run() (/home/felipeo/code/arrow/cpp/ninja/debug/libarrow_testing.so.1600+0xd84b4b) (BuildId: dd9af0bafdb1786050262e8f6002568a9f08ecf6) apache#14 0x5608ee54506d in RUN_ALL_TESTS() /usr/include/gtest/gtest.h:2490:46 apache#15 0x5608ee544fb9 in main /home/felipeo/code/arrow/cpp/src/arrow/util/logging_test.cc:129:10 apache#16 0x7fd93de29d8f in __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16 apache#17 0x7fd93de29e3f in __libc_start_main csu/../csu/libc-start.c:392:3 apache#18 0x5608ed7b4f64 in _start (/home/felipeo/code/arrow/cpp/ninja/debug/arrow-utility-test+0x1224f64) (BuildId: 81cfdc36b7a960a7249ecd5884beaa869140ab89) Address 0x7fff1804c48f is located in stack of thread T0 at offset 399 in frame #0 0x5608ee05dc9f in decltype(std::declval<arrow::StringAppender&>()(std::basic_string_view<char, std::char_traits<char> >{})) arrow::internal::StringFormatter<arrow::TimestampType, void>::operator()<std::chrono::duration<long, std::ratio<1l, 1000l> >, arrow::StringAppender&>(std::chrono::duration<long, std::ratio<1l, 1000l> >, long, arrow::StringAppender&) /home/felipeo/code/arrow/cpp/src/arrow/util/formatting.h :486

…zone (#45051) ### Rationale for this change If the timezone database is present on the system, but does not contain a timezone referenced in a ORC file, the ORC reader will crash with an uncaught C++ exception. This can happen for example on Ubuntu 24.04 where some timezone aliases have been removed from the main `tzdata` package to a `tzdata-legacy` package. If `tzdata-legacy` is not installed, trying to read a ORC file that references e.g. the "US/Pacific" timezone would crash. Here is a backtrace excerpt: ``` #12 0x00007f1a3ce23a55 in std::terminate() () from /lib/x86_64-linux-gnu/libstdc++.so.6 #13 0x00007f1a3ce39391 in __cxa_throw () from /lib/x86_64-linux-gnu/libstdc++.so.6 #14 0x00007f1a3f4accc4 in orc::loadTZDB(std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) () from /tmp/arrow-HEAD.ArqTs/venv-wheel-3.12-manylinux_2_17_x86_64.manylinux2014_x86_64/lib/python3.12/site-packages/pyarrow/libarrow.so.1900 #15 0x00007f1a3f4ad392 in std::call_once<orc::LazyTimezone::getImpl() const::{lambda()#1}>(std::once_flag&, orc::LazyTimezone::getImpl() const::{lambda()#1}&&)::{lambda()#2}::_FUN() () from /tmp/arrow-HEAD.ArqTs/venv-wheel-3.12-manylinux_2_17_x86_64.manylinux2014_x86_64/lib/python3.12/site-packages/pyarrow/libarrow.so.1900 #16 0x00007f1a4298bec3 in __pthread_once_slow (once_control=0xa5ca7c8, init_routine=0x7f1a3ce69420 <__once_proxy>) at ./nptl/pthread_once.c:116 #17 0x00007f1a3f4a9ad0 in orc::LazyTimezone::getEpoch() const () from /tmp/arrow-HEAD.ArqTs/venv-wheel-3.12-manylinux_2_17_x86_64.manylinux2014_x86_64/lib/python3.12/site-packages/pyarrow/libarrow.so.1900 #18 0x00007f1a3f4e76b1 in orc::TimestampColumnReader::TimestampColumnReader(orc::Type const&, orc::StripeStreams&, bool) () from /tmp/arrow-HEAD.ArqTs/venv-wheel-3.12-manylinux_2_17_x86_64.manylinux2014_x86_64/lib/python3.12/site-packages/pyarrow/libarrow.so.1900 #19 0x00007f1a3f4e84ad in orc::buildReader(orc::Type const&, orc::StripeStreams&, bool, bool, bool) () from /tmp/arrow-HEAD.ArqTs/venv-wheel-3.12-manylinux_2_17_x86_64.manylinux2014_x86_64/lib/python3.12/site-packages/pyarrow/libarrow.so.1900 #20 0x00007f1a3f4e8dd7 in orc::StructColumnReader::StructColumnReader(orc::Type const&, orc::StripeStreams&, bool, bool) () from /tmp/arrow-HEAD.ArqTs/venv-wheel-3.12-manylinux_2_17_x86_64.manylinux2014_x86_64/lib/python3.12/site-packages/pyarrow/libarrow.so.1900 #21 0x00007f1a3f4e8532 in orc::buildReader(orc::Type const&, orc::StripeStreams&, bool, bool, bool) () from /tmp/arrow-HEAD.ArqTs/venv-wheel-3.12-manylinux_2_17_x86_64.manylinux2014_x86_64/lib/python3.12/site-packages/pyarrow/libarrow.so.1900 #22 0x00007f1a3f4925e9 in orc::RowReaderImpl::startNextStripe() () from /tmp/arrow-HEAD.ArqTs/venv-wheel-3.12-manylinux_2_17_x86_64.manylinux2014_x86_64/lib/python3.12/site-packages/pyarrow/libarrow.so.1900 #23 0x00007f1a3f492c9d in orc::RowReaderImpl::next(orc::ColumnVectorBatch&) () from /tmp/arrow-HEAD.ArqTs/venv-wheel-3.12-manylinux_2_17_x86_64.manylinux2014_x86_64/lib/python3.12/site-packages/pyarrow/libarrow.so.1900 #24 0x00007f1a3e6b251f in arrow::adapters::orc::ORCFileReader::Impl::ReadBatch(orc::RowReaderOptions const&, std::shared_ptr<arrow::Schema> const&, long) () from /tmp/arrow-HEAD.ArqTs/venv-wheel-3.12-manylinux_2_17_x86_64.manylinux2014_x86_64/lib/python3.12/site-packages/pyarrow/libarrow.so.1900 ``` ### What changes are included in this PR? Catch C++ exceptions when iterating ORC batches instead of letting them slip through. ### Are these changes tested? Yes. ### Are there any user-facing changes? No. * GitHub Issue: #40633 Authored-by: Antoine Pitrou <[email protected]> Signed-off-by: Sutou Kouhei <[email protected]>

ARROW-5: Fix error when run maven install

89b6a03

lw-lin added 2 commits February 20, 2016 09:54

Update drill-fmpp-maven-plugin to 1.5.0

7374717

Remove fmpp/freemarker version numbers

b8d16d5

lw-lin changed the title ~~ARROW-5: Fix error when run maven install~~ ARROW-5: Update drill-fmpp-maven-plugin to 1.5.0 and remove fmpp/freemarker version numbers Feb 20, 2016

asfgit closed this in 7e76e3a Feb 20, 2016

lw-lin deleted the ARROW-5 branch February 20, 2016 08:22

lw-lin changed the title ~~ARROW-5: Update drill-fmpp-maven-plugin to 1.5.0 and remove fmpp/freemarker version numbers~~ ARROW-5: Update drill-fmpp-maven-plugin to 1.5.0 Feb 21, 2016

vgoklani mentioned this pull request Dec 26, 2017

pyarrow: Segmentation fault (core dumped) #1450

Closed

robambalu mentioned this pull request Apr 12, 2018

pyarrow crash on converstion of Pandas dataframe -> arrow with Decimal column #1888

Closed

songqing added a commit to songqing/arrow that referenced this pull request Jun 6, 2018

install plasma_queue.h (apache#1)

84e8497

dragan224 mentioned this pull request Jul 18, 2018

Using arrow to write parquet (ends up with an empty table) #2287

Closed

icexelloss referenced this pull request in icexelloss/arrow Mar 6, 2023

Merge pull request #1 from icexelloss/apacheGH-32884-ordered-agg

5ac8c65

PR comments from apache#34311

sunchao added a commit to sunchao/arrow that referenced this pull request Mar 17, 2023

Fix valueCount when loading NullVector (apache#1)

eb033fd

We should set `valueCount` when loading `NullVector`. Unfortunately there's no `nullCount` field.

icexelloss referenced this pull request in icexelloss/arrow Mar 31, 2023

Merge pull request #1 from westonpace/refactor_10

fb9d73d

Minor fixes to repair AVX2 build

hqx871 mentioned this pull request May 16, 2023

[c++] arrow::int32 throws exc_bad_access #35616

Closed

asfimport mentioned this pull request Jun 6, 2023

[Python] After upgrade pyarrow from 0.15 to 0.17.1 connect to hdfs don`t work with libdfs jni #17210

Closed

arthurpassos mentioned this pull request Jun 9, 2023

GH-32723: [C++][Parquet] Add option to use LARGE* variants of binary types #35825

Closed

asfimport mentioned this pull request Jul 19, 2023

[R] Troubles with using augmented columns #33464

Open

felipecrv added a commit to felipecrv/arrow that referenced this pull request Jul 19, 2023

Fix issue apache#1

40ef0ba

This is ugly, but necessary because assuming CleanOffsets always produces a Buffer that can be sliced by offsets.offset() is wrong.

CurtHagenlocher mentioned this pull request Jul 21, 2023

[C#] Using C API fails at runtime when targeting .NET 4.7.2 #36812

Closed

geyungjen mentioned this pull request Aug 7, 2023

arrow::flight::FlightClient::AuthenticateBasicToken C++ Segmentation fault (core dumped) #37033

Closed

allinux mentioned this pull request Aug 25, 2023

[C++] Valgrind detect memory leak when using s3fs(InitializeS3/FinalizeS3) #37375

Closed

jorisvandenbossche mentioned this pull request Oct 6, 2023

GH-38068: [C++][CI] Fixing Parquet unittest arrow_reader_writer_test.cc compile #38069

Merged

niyue mentioned this pull request Oct 13, 2023

GH-37753: [C++][Gandiva] Add external function registry support #38116

Merged

Matan2t mentioned this pull request Nov 19, 2023

[C++] segmentation fault in s_aws_logger_redirect_get_log_level #38788

Open

pitrou mentioned this pull request Dec 4, 2023

GH-33984: [C++][Python] DLPack implementation for Arrow Arrays (producer) #38472

Merged

asfimport mentioned this pull request Oct 11, 2022

[Python] Coredump when joining big large_strings #33151

Open

niyue mentioned this pull request Dec 9, 2023

GH-37848: [C++][Gandiva] Migrate LLVM JIT engine from MCJIT to ORC v2/LLJIT #39098

Merged

zeroshade pushed a commit to zeroshade/arrow that referenced this pull request Dec 13, 2023

Merge pull request apache#1 from zeroshade/http-examples/go

ba3b5c0

prniii mentioned this pull request Jan 12, 2024

[Python] macOS Segfault on Import, Both arm64 and x86_64 #37010

Open

av8or1 mentioned this pull request Jan 18, 2024

GH-38703: [C++][FS][Azure] Implement DeleteFile() #39648

Closed

mdavis-xyz mentioned this pull request Jan 29, 2024

[Python] Parsing timestamp with microsecond in CSV fails even with %f #39839

Open

gbronner mentioned this pull request Apr 22, 2024

[C++] ReadNext in arrow::RecordBatchReader returns invalid status on second or subsequent items #41339

Closed

asfimport mentioned this pull request Oct 13, 2022

[Plasma][C++] plasma-store-server start with -h , occur bus error #33204

Closed

ripplehang mentioned this pull request Aug 23, 2024

GH-43535: [C++] Support the AWS S3 SSE-C encryption #43601

Merged

Tjev mentioned this pull request Sep 30, 2024

[Python][FS][Azure] C++ Exceptions leaking from PyArrow #44269

Closed

yuqi1129 mentioned this pull request Oct 16, 2024

[Python] Read GCS parquet file failed with pyarrow #44438

Open

Voltagabbana mentioned this pull request Oct 18, 2024

Memory Leak when Importing Parquet File with PyArrow Engine in Pandas #44472

Open

asfimport mentioned this pull request Jan 10, 2023

[Java] Remove reader index and writer index from ArrowBuf apache/arrow-java#272

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ARROW-5: Update drill-fmpp-maven-plugin to 1.5.0 #1

ARROW-5: Update drill-fmpp-maven-plugin to 1.5.0 #1

lw-lin commented Feb 19, 2016

lw-lin commented Feb 19, 2016

jacques-n commented Feb 19, 2016

lw-lin commented Feb 20, 2016

jacques-n commented Feb 20, 2016

lw-lin commented Feb 20, 2016

jacques-n commented Feb 20, 2016

lw-lin commented Feb 20, 2016

AllenFang commented Feb 20, 2016

ARROW-5: Update drill-fmpp-maven-plugin to 1.5.0 #1

ARROW-5: Update drill-fmpp-maven-plugin to 1.5.0 #1

Conversation

lw-lin commented Feb 19, 2016

lw-lin commented Feb 19, 2016

jacques-n commented Feb 19, 2016

lw-lin commented Feb 20, 2016

jacques-n commented Feb 20, 2016

lw-lin commented Feb 20, 2016

jacques-n commented Feb 20, 2016

lw-lin commented Feb 20, 2016

AllenFang commented Feb 20, 2016