diff --git a/wiki/Hacking.md b/wiki/Hacking.md new file mode 100644 index 0000000..3ca3b8e --- /dev/null +++ b/wiki/Hacking.md @@ -0,0 +1,6 @@ +# HACKING + +There is information +about contributing to qpdf in the [manual](https://qpdf.readthedocs.io/en/latest/contributing.html) +that covers the information that used to be on this page. As questions come up, I may add discussion +here, particular as things are incubating. diff --git a/wiki/Home.md b/wiki/Home.md new file mode 100644 index 0000000..0a8bc90 --- /dev/null +++ b/wiki/Home.md @@ -0,0 +1,20 @@ +# Welcome to the qpdf wiki! + +The intention is to use this wiki for information that may be useful but hasn't made it into the +main documentation. Please see the list of pages on the sidebar. + +The wiki also provides some information about recent and planned changes, including changes planned +for qpdf 12 and the next steps in the [[pages epic|Pages epic : next steps]], the work to support +more document-level structures (such as the preservation of outlines) and page level +transformations (such as scaling). The information either has not made it into the manual yet, or +goes into more detail than is provided in the manual and release notes. + +## Planned changes + +### Treatment of uninitialized QPDFObjectHandles and null objects + +Changes to the treatment of uninitialized object handles and null objects are planned for both qpdf +11.10 and qpdf 12. for details see + +- [[PDF null objects vs qpdf null objects]] +- [[Use of default constructed object handles in qpdf to indicate failure or error]] diff --git a/wiki/PDF-null-objects-vs-qpdf-null-objects.md b/wiki/PDF-null-objects-vs-qpdf-null-objects.md new file mode 100644 index 0000000..96d7242 --- /dev/null +++ b/wiki/PDF-null-objects-vs-qpdf-null-objects.md @@ -0,0 +1,67 @@ +The PDF specification states + +> The null object has a type and value that are unequal to those of any other object. There shall be +> only one object of type null, denoted by the keyword null. + +This is straightforward - there is a single object and, by implication, it is immutable. In other +words, all nulls encountered are equal. + +Unfortunately, in qpdf the situation is not quite as straightforward. There are a couple of reasons +for this: + +- qpdf objects have a variety of diagnostic information attached to them. For example, an attempt to + retrieve an object with `getKey("/Blah")` from an array will return a null object that carries the + description " -> null returned from getting key /Blah from non-Dictionary". + +- For performance reasons, qpdf does not have a distinct 'indirect reference' type. Under the hood, + all object handles are references (or rather shared pointers) to the actual object. To avoid a + further level of indirection, qpdf objects have an id and generation attached; if id and + generation are both zero, the object is a direct objects. Otherwise, it is an indirect reference. + As a result, in qpdf each indirect null is a distinct object. Furthermore, qpdf allows indirect + objects to be updated with the `replaceObject` method. It also allows direct objects, including + direct nulls, to be made indirect. + +Therefore, in qpdf null objects are not all the same, nor are they immutable. This is particularly +true for indirect nulls. For example, if a foreign object (i.e. an object from a different PDF +file / QPDF object) is copied using the `copyForeignObject` method, any references to pages that +have not already been copied will appear as an indirect null. If the page is subsequently copied, +those indirect nulls will be transparently updated to become the copied page. + +On the other hand, qpdf has to create a large number of nulls behind the scenes, most of which are +short-lived and never seen by users of the qpdf library. To avoid the overhead of creating these +nulls, qpdf uses references to a single shared null object. This does not normally cause any +problems, but it could if an attempt was made to mutate such a shared null. + +qpdf 12 will make a clearer distinction between these two types of null objects - there will be +**mutable nulls** and **shared nulls**. In almost all existing situations, both types of null object +will behave identically. The main differences are that + +- shared nulls cannot be made direct, +- shared nulls never have an owner (i.e. `getOwningQPDF` will always return a nullptr), and +- attempts to set descriptions, offsets, etc. will be ignored by shared nulls. + +To implement shared nulls, qpdf 12 will use default constructed or uninitialized object handles. The +`isInitialized` method will be removed, and for other methods such as `isNull`, `unparse`, +`shallowCopy`, etc, uninitialized object handles will be treated as null objects. + +Shared nulls can be distinguished from mutable nulls (or any other object) by converting them to +a `bool` value - shared nulls will evaluate to `false`, all other objects to `true`. + +Existing code that uses the `isInitialized` method can normally be updated by simply updating method +calls such as `some_object_handle.isInitialized()` with the object handle +`some_object_handle` itself. This will work when used as condition in if and while statements, in +logical expressions or when initializing a bool variable. In some other situations, such as when +assigning to an existing bool variable or in a return statement it will be necessary to explicitly +cast the object handle to bool. + +The only complication arises when `some_object_handle` may be a null object that needs to be treated +differently from an uninitialized object. This is an unusual situation and the only circumstance +where this is likely to occur is when a user method returns an uninitialized object handle to signal +some sort of failure. In this situation it is necessary to consider whether a legitimate null object +could possibly be a shared null (objects returned by `newNull` or `getKey` will never be shared +nulls), and if so, replace such nulls with a freshly constructed null using a call to +`newNull()`. An alternative solution would be to use `std::optional` as the return +type / variable type, which would distinguish between a missing object handle and an uninitialized +object handle. + +(To comment, please go to the [discussion](https://github.com/qpdf/qpdf/discussions/xxxx) diff --git a/wiki/Pages-epic-:-next-steps.md b/wiki/Pages-epic-:-next-steps.md new file mode 100644 index 0000000..cce526f --- /dev/null +++ b/wiki/Pages-epic-:-next-steps.md @@ -0,0 +1,100 @@ +The current plan is to implement the following steps of +the [pages epic](https://github.com/qpdf/qpdf/discussions/1104) next. The plan is provisional and +subject to change. Comments are welcome [here](https://github.com/qpdf/qpdf/discussions/xxx) . The +nitty-gritty implementation detail is discussed +at [qpdf/qpdf-dev](https://github.com/qpdf/qpdf-dev/discussions/3) . + +Further detail will be added as the work progresses. + +## Step 1: add new `--input` option + +The plan is to provide a new option to allow all the input files to be defined outside the +`--pages` option. So, instead of + +``` +qpdf main.pdf \ + --pages . --file=input1.pdf --range=3-z --file=input2.pdf --password=xyz --file=input3.pdf -- \ + out.pdf +``` + +it will be possible to use + +``` +qpdf --input --file=main.pdf --file=input1.pdf --file=input2.pdf --password=xyz --file=input3.pdf -- \ + --pages --id=0 --id=1 --range=3-z --id=2 -- \ + out.pdf +``` + +instead. While the new option provides no benefit in this example, it will make it easier to deal +with complex assemblies of pages where the same input file appears multiple times in the `--pages` +options. More importantly, it provides the following additional benefits: + +- It will provide a convenient way to add options that apply to an entire input file. For an example + see the `--destinations` option described under + [step 2b](#step-2b-add-option-to-preserve-named-destinations-when-merging-files). +- It will be more convenient (and more efficient) for users of job JSON or the QPDFJob C++/C API. + Such users are likely to have a table of inputs already, and in many cases each entry in + the `--pages` option will be for a single page. +- It will provide an opportunity to enhance the QPDFJob interface in future to allow additional + types of inputs to be specified in the `--files` option, including memory buffers or QPDF objects. + +In order to make the commands less verbose, short-forms are planned for `--file`, `--password`, +`--range` and `--id`, allowing the examples given above to be abbreviated to + +``` +qpdf main.pdf \ + --pages . -f=input1.pdf -r=3-z -f=input2.pdf -p=xyz -f=input3.pdf -- \ + out.pdf +``` + +and + +``` +qpdf --input -f=main.pdf -f=input1.pdf -f=input2.pdf -p=xyz -f=input3.pdf -- \ + --pages -i=0 -i=1 -r=3-z -i=2 i=3 -- \ + out.pdf +``` + +## step 2a: allow rotation to be specified inside the `--pages` option + +For example, to combine two input files with the last page of each file rotated by 90 degrees, +instead of writing + +``` +qpdf main.pdf \ + --pages . --file=input1.pdf --range=3-z-- \ + --rotate=90:3,27 \ + out.pdf +``` + +it will be possible to write + +``` +qpdf --input -f=main.pdf -f=input1.pdf -- \ + --pages + -i=0 -r=1-r2 \ + -i=0 -r=z --rotate=90 \ + -i=1 -r=1-r2 \ + -i=1 -r=z --rotate=90 -- \ + out.pdf +``` + +This avoids having to work out the page number (in the output file) of the final page of each input +file. In particular, if this was a monthly job where the length of the input files may vary, it +would avoid the need to adjust the script every month. + +It will also be more convenient for users of the QPDFJob interface. + +The plan is for future modification options (e.g. scaling or cropping) to take the same approach. + +## step 2b: add option to preserve named destinations when merging files + +The plan is to have a new options that allows for named destinations to be preserved, which will +allow more hyperlinks in merged files to work correctly. Preservation of named destinations is also +a prerequisite for a future option to preserve outlines. + +There is a runtime cost involved, and therefore preservation is going to be optional. It will be +possible to select the option globally or on a per input file basis. + +In order to avoid name clashes, it is currently planned to prepend a unique id for each input file +to each destination name. diff --git a/wiki/Use-of-default-constructed-object-handles-in-qpdf-to-indicate-failure-or-error.md b/wiki/Use-of-default-constructed-object-handles-in-qpdf-to-indicate-failure-or-error.md new file mode 100644 index 0000000..8bb0ff4 --- /dev/null +++ b/wiki/Use-of-default-constructed-object-handles-in-qpdf-to-indicate-failure-or-error.md @@ -0,0 +1,56 @@ +It is expected that qpdf 11.10 will include an operator to convert object handles to bool. +Uninitialized / default constructed object handles will evaluate to false. This note outlines how +this operator can be used. + +For the basic scenario, take `some_method` that returns an object handle: + +```c++ +QPDFObjectHandle some_method() +{ + // do some calculations + if (success) { + return some_valid_oh; + } else { + return {}; + } +} +``` + +The operator can now be used for error handling: + +```c++ +if (auto result = some_method()) { + // handle success +} else { + // handle failure +} +``` + +This can be expanded to multiple method calls: + +```c++ +auto part1 = some_method(); +auto part2 = some_other_method(); +if (part1 && part2 && other_relevant_conditions) { + // handle success +} else { + // handle failure +} +``` + +If the method is run for side effects only, we can use it to initialize a bool variable: + +```c++ +bool success{some_method()}; +``` + +Note however that in many other situation it will be necessary to explicitly convert the object +handle to bool, notably when assigning to an existing variable or using it as a bool return value + +```c++ +existing_flag = static_cast(some_method()); +``` + +**Important** Methods do not normally return null objects when successful. If it is possible that a +null object will be returned, it is important to be aware of planned changes to null objects +described at [[PDF null objects vs qpdf null objects]] diff --git a/wiki/_sidebar.md b/wiki/_sidebar.md new file mode 100644 index 0000000..3d9fe42 --- /dev/null +++ b/wiki/_sidebar.md @@ -0,0 +1,8 @@ +- [[Home]] + +- [[Planned Changes|Home#Planned-Changes]] + +- [[Pages epic : next steps]] + +- [[qpdf Wrappers]] +- [[Hacking]] diff --git a/wiki/null.md b/wiki/null.md new file mode 100644 index 0000000..58b2188 --- /dev/null +++ b/wiki/null.md @@ -0,0 +1,70 @@ +PDF null objects vs qpdf null objects +===================================== + +The PDF specification states + +> The null object has a type and value that are unequal to those of any other object. There shall be +> only one object of type null, denoted by the keyword null. + +This is straightforward - there is a single object and, by implication, it is immutable. In other +words, all nulls encountered are equal. + +Unfortunately, in qpdf the situation is not quite as straightforward. There are a couple of reasons +for this: + +- qpdf objects have a variety of diagnostic information attached to them. For example, an attempt to + retrieve a object with `getKey("/Blah")` from an array will return a null object that carries the + description " -> null returned from getting key /Blah from non-Dictionary". + +- For performance reasons, qpdf does not have a distinct 'indirect reference' type. Under the hood, + all object handles are references (or rather shared pointers) to the actual object. To avoid a + further level of indirection, qpdf objects have an id and generation attached; if id and + generation are both zero, the object is a direct objects. Otherwise, it is an indirect reference. + As a result, in qpdf each indirect null is a distinct object. Furthermore, qpdf allows indirect + objects to be updated with the `replaceObject` method. It also allows direct objects, including + direct nulls, to be made indirect. + +Therefore, in qpdf null objects are not all the same, nor are they immutable. This is particularly +true for indirect nulls. For example, if a foreign object (i.e. an object from a different PDF +file / QPDF object) is copied using the `copyForeignObject` method, any references to pages that +have not already been copied will appear as an indirect null. If the page is subsequently copied, +those indirect nulls will be transparently updated to become the copied page. + +On the other hand, qpdf has to create a large number of nulls behind the scenes, most of which are +short-lived and never seen by users of the qpdf library. To avoid the overhead of creating these +nulls, qpdf uses references to a single shared null object. This does not normally cause any +problems, but it could if an attempt was made to mutate such a shared null. + +qpdf 12 will make a clearer distinction between these two types of null objects - there will be +**mutable nulls** and **shared nulls**. In allmost all existing situations, both types of null +object will behave identically. The main differences are that + +- shared nulls cannot be made direct, +- shared nulls never have an owner (i.e. `getOwningQPDF` will always return a nullptr), and +- attempts to set descriptions, offsets, etc will be ignored by shared nulls. + +To implement shared nulls, qpdf 12 will use default constructed or uninitialized object handles. The +`isInitialized` method will be removed, and for other methods such as `isNull`, `unparse`, +`shallowCopy`, etc, uninitialized object handles will be treated as null objects. + +Shared nulls can be distinguished from mutable nulls (or any other object) by converting them to +a `bool` value - shared nulls will evaluate to `false`, all other objects to `true`. + +Existing code that uses the `isInitialized` method can normaly be updated by simply updating method +calls such as `some_object_handle.isInitialized()` with the object handle +`some_object_handle` itself. This will work when used as condition in if and while statements, in +logical expressions or when initializing a bool variable. In some other situations, such as when +assigning to an existing bool variable or in a return statement it will be necessary to explicitely +cast the object handle to bool. + +The only complication arises when `some_object_handle` may be a null object that needs to be treated +differently from an uninitialized object. This is an unusual situation and the only circumstance +where this is likely to occur is when a user method returns an uninitialized object handle to +signal some sort of failure. In this situation it is necessary to consider whether a legitimate null +object could possibly be a shared null (objects returned by `newNull` or `getKey` will never be +shared bulls), and if so, replace such nulls with a freshly constructed null using a call to +`newNull()`. An alternative solution would be to use `std::optional` as the +return type / variable type, which would distinguish between a missing object handle and an +uninitialized object handle. + +(To comment, please go to https://github.com/qpdf/qpdf/discussions/xxxx) diff --git a/wiki/qpdf-Wrappers.md b/wiki/qpdf-Wrappers.md new file mode 100644 index 0000000..3ba44d5 --- /dev/null +++ b/wiki/qpdf-Wrappers.md @@ -0,0 +1,17 @@ +# Wrappers around qpdf + +This page includes a list of known wrappers around qpdf. If you have a library that enhances qpdf or +exposes its functionality for programmers of languages other than C++, let me know, and I'll add it +to this list. + +## Python + +* [pikepdf](https://pypi.org/project/pikepdf/): a fully featured Python PDF library build upon qpdf. + It exposes most of the functionality of qpdf in a "Pythonic" (idiomatically Python) way and offers + many higher-level interfaces to work with PDF files beyond what is provided directly by qpdf. + +## C# + +* [QPdfNet](https://github.com/Sicos1977/QPdfNet): a C# package that exposes uses qpdf's `QPDFJob` + API to expose functionality that is available from the qpdf CLI without having to actually invoke + the `qpdf` command. diff --git a/wiki/uninit.md b/wiki/uninit.md new file mode 100644 index 0000000..3cd9d60 --- /dev/null +++ b/wiki/uninit.md @@ -0,0 +1,58 @@ +# Use of default constructed object handles in qpdf to indicate failure / error + +It is expected that qpdf 11.10 will include an operator to convert object handles to bool. +Uninitialized / default constructed object handles will evaluate to false. This note outlines how +this operator can be used. + +For the basic scenario, take `some_method` that returns an object handle: + +```c++ +QPDFObjectHandle some_method() +{ + // do some calculations + if (success) + return some_valid_oh; + else + return {}; +} +``` + +The operator can now be used for error handling: + +```c++ +if (auto result = some_method()) { + // handle success +} else { + // handle failure +} +``` + +This can be expanded to multiple method calls: + +```c++ +auto part1 = some_method(); +auto part2 = some_other_method(); +if (part1 && part2 && other_relevant_conditions) { + // handle success +} else { + // handle failure +} +``` + +If the method is run for side effects only, we can use it to initialize a bool variable: + +```c++ +bool success{some_method()}; +``` + +Note however that in many other situation it will be necessary to explicitly convert the object +handle to bool, notably when assigning to an existing variable or using it as a bool return value + +```c++ +existing_flag = static_cast(some_method()); +``` + +**Important** Methods do not normally return null objects when successful. If it is possible that +a null object will be returned, it is important to be aware of planned changes to null objects +described at https://github.com/qpdf/qpdf/wiki/null.md +