Add README for updating PCRE

1 files changed, 733 insertions, 0 deletions

diff --git a/erts/emulator/pcre/README.pcre_update.md b/erts/emulator/pcre/README.pcre_update.md
new file mode 100644
index 0000000000..5f2414f0d9
--- /dev/null
+++ b/erts/emulator/pcre/README.pcre_update.md
@@ -0,0 +1,733 @@
+# How to update the PCRE version used by Erlang
+
+## The basic changes to the PCRE library
+
+To work with the Erlang VM, PCRE has been changed in two important ways:
+
+1. The main execution machine in pcre\_exec has been modified so that
+matching can be interrupted and restarted. This functionality utilizes
+the code that implements recursion by allocating explicit
+"stack-frames" in heap space, which basically means that all local
+variables in the loop are part of a struct which is kept in "malloced"
+memory on the heap and there are no real stack variables that need to
+be pushed on the C stack in the case of recursive calls. This is a
+technique we also use inside the VM to avoid building large C
+stacks. In PCRE this is enabled by the NO\_RECURSE define, so that is a
+prerequisite for the ERLANG\_INTEGRATION define which also adds labels
+at restart points and counts "reductions".
+
+2. All visible symbols in PCRE gets the erts_ prefix, so that NIF's
+and such using a "real" pcre library does not get confused (or 're'
+gets confused when a "real" pcre library get's loaded into the VM
+process).
+
+3. All irrelevant functionality has been stripped from the library,
+which means for example UTF16 support, jit, DFA execution
+etc. Basically the source files handling this are removed, together
+with any build support from the PCRE project. We have our own
+makefiles etc.
+
+## Setting up an environment for the work
+
+I work with four temporary directories when doing this (the examples
+are from the updating of pcre-7.6 to pcre-8.33);
+
+       ~/tmp/pcre> ls
+       epcre-7.6  epcre-8.33  pcre-7.6  pcre-8.33
+
+I've unpacked the plain pcre sources in pcre-* and will work with our
+patched sources in the epcre-* directories.
+
+Make sure your ERL_TOP contains a *built* version of Erlang (and you have made a branch)
+
+First unpack the pcre libraries (which will create the pcre-*
+directories) and then copy our code to the old epcre directory:
+
+      ~/tmp/pcre> tar jxf $ERL_TOP/erts/emulator/pcre/pcre-7.6.tar.bz2 
+      ~/tmp/pcre> tar jxf ~/Downloads/pcre-8.33.tar.bz2 
+      ~/tmp/pcre> mkdir epcre-7.6  epcre-8.33
+      ~/tmp/pcre> cd epcre-7.6/
+      ~/tmp/pcre/epcre-7.6> cp -r $ERL_TOP/erts/emulator/pcre/* .
+      ~/tmp/pcre/epcre-7.6> rm pcre-7.6.tar.bz2
+
+Leave the obj directory, you may need the libepcre.a file...
+
+If you find it easier, you can revert the commit in GIT that adds the
+erts_ prefix to the previous version before continuing work, but as
+that is a quite small diff in newer versions of PCRE, it is probably
+not worth it. Still, you will find the erts_ prefix being a separate
+commit when integrating 8.33, so if you're nice, you will do the same
+for the person coming after you...
+
+## Generating a diff for our changes to PCRE
+
+Before you generate a diff (that, in an ideal world, would be used to
+automatically patch the newer version of pcre, which will probably
+only work for minor PCRE updates), we need to configure the old pcre.
+
+       ~/tmp/pcre/epcre-7.6> cd ../pcre-7.6
+       ~/tmp/pcre/pcre-7.6> ./configure --enable-utf8 --enable-unicode-properties --disable-shared --disable-stack-for-recursion
+
+Note that for newer versions, the configure flag '--enable-utf8'
+should be replaced with '--enable-utf'
+
+So we now generate a diff:
+
+    ~/tmp/pcre/pcre-7.6> cd ../epcre-7.6
+   ~/tmp/pcre/epcre-7.6> (for x in *.[ch]; do if [ -f ../pcre-7.6/$x ]; then diff -c ../pcre-7.6/$x $x; fi; done ) > ../epcre-7.6_clean.diff
+
+### What the diff means
+
+Let's now walk through the relevant parts of the diff. Some of the
+differences might come from patches that probably are already in the
+new version, For example in out 7.6, we had a security patch which
+added the define WORK_SIZE_CHECK and used it in some places. Those can
+probably safely be ignored, but to be on the safe side, check what's
+already integrated in the new version.
+
+The interesting part is in pcre_exec.c. You will see things like 
+
+    #ifdef ERLANG_INTEGRATION
+    ...
+    #endif
+
+or
+ 
+    #if defined(ERLANG_INTEGRATION)
+    ...
+    #endif
+
+and a lot of
+
+    COST_CHK(1);
+
+or
+
+    COST(min);
+
+and
+
+    /* LOOP_COUNT: Ok */
+    /* LOOP_COUNT: CHK */
+    /* LOOP_COUNT: COST */
+	
+
+scattered over the main loop. Those mean the following:
+
+* COST(int x) - consume reductions proportional to the integer
+  parameter, but no need for interruption here (it's like
+  bump_reductions without trapping). The loop they apply to also has a
+  'LOOP_COUNT: COST' comment at it's head.
+
+* COST\_CHK(int x) - like COST(x), but also check that the reduction
+counter does not reach zero. If it does, leave the execution loop to
+be restarted at a later point. No real stack variables can be live
+here. Note that variables like 'max' and 'min' are *not* real stack
+variables, the NO\_RECURSION setting has taken care of that. 'i' is a
+stack variable that's explicitly saved when trapping, so that will
+also be correct when returning from a trap. So will 'c', 'rrc' and
+flags like 'utf8', 'minimize' and 'posessive'. Those can also be
+regarded as "non C-stack variables". The loop where they reside also
+has a 'LOOP\_COUNT: CHK' comment.
+
+* /* LOOP_COUNT: Ok */ - means that I have checked the loop and it
+  only runs a deterministic set of iterations regardless of input, or
+  it has a call to RRECURSE in it's body, why we need not add more
+  cost than the normal reduction counting that will occur for each
+  instruction demands.
+
+The thing is that each loop in the function 'match' should be marked
+with one of these comments. If no comment is present after you patched
+the new release (if you successfully manage to do it automatically),
+it may be a new regexp instruction that is added since the last
+release.
+
+You will need to manually go through the main 'match' loop after
+upgrading to verify that there are no unhandled loops in the regexp
+machine loop (!).
+
+The COST\_CHK macro works like this:
+
+1. Add to the loop count.
+2. If loop count > limit:
+    1. Store the line (+100) in the Xwhere member of the frame structure
+    2. Goto LOOP\_COUNT\_BREAK, which ultimately returns from the function
+3. Insert a label, which is named L\_LOOP\_COUNT\_<line number>
+
+LOOP\_COUNT\_BREAK code will create an extra "stack frame" on the heap
+allocated stack used if NO\_RECURSION is set, and will store the few
+locals that are not already in the ordinary stack frame there (like
+'c' and 'i').
+
+When we continue execution (after a trap up to the main Erlang
+scheduler), we will jump to LOOP\_COUNT\_RETURN, which will restore
+the local variables and will jump to the labels. The jump code looks
+like this in the C source:
+
+     	switch (frame->Xwhere) 
+      	       {
+      #include "pcre_exec_loop_break_cases.inc"
+      	       default:
+		        DPRINTF(("jump error in pcre match: label %d non-existent\n", frame->Xwhere));
+        		return PCRE_ERROR_INTERNAL;
+      		}
+
+When building, pcre\_exec\_loop\_break\_cases.inc will be generated
+during build by pcre.mk, it will look like:
+
+     case 791: goto L_LOOP_COUNT_691;
+     case 1892: goto L_LOOP_COUNT_1792;
+     case 1999: goto L_LOOP_COUNT_1899;
+
+etc
+
+So, simply put, all C-stack variables are saved when we have consumed
+our reductions, we return from the function and, as there is no real
+recursion we immediately fall out into the re:run BIF, which with the
+help of a magic binary keeps track of the heap allocated stack for the
+regexp machine. When we return from trapping out to the scheduler, all
+vital data is restored and we continue from exactly the same state as
+we left. What's needed is to patch this into the new pcre_exec and
+check all new instructions to determine what might need updating in
+terms of COST, COST\_CHK etc.
+
+Well, that's *almost* everything, because there is of course more...
+
+The actual interface function, 'pcre\_exec', needs the same treatment
+as the actual regexp machine loop, that is we need to store all local
+variables between restarts. Unfortunately the NO\_RECURSE setting does
+not do this, we need to do it ourselves. So there's quite a diff in
+that function too, where a big struct is declared, containing every
+local variable in that function, together with either local copies
+that are swapped in and out, or macros that directly access the heap
+allocated struct. The struct is called `PcreExecContext`.
+
+If a context is present, we are restarting and therefore restore
+everything. If we are restarting we can also skip all initialization
+code in the function and jump more or less directly to the
+RESTART_INTERRUPTED label and the call to 'match', which is the actual
+regexp machine loop.
+
+There are a few places in the pcre_exec we need to do some housekeeping, you will see code like:
+
+ 	if ((extra_data->flags & PCRE_EXTRA_LOOP_LIMIT) != 0) 
+ 	  {
+		*extra_data->loop_counter_return = 
+ 	    		(extra_data->loop_limit - md->loop_limit);
+          }
+
+Make sure, after updating, that this housekeeping is done whenever we
+do not reach the call to 'match'.
+
+So, now we in theory know what to do, so let's do it:
+
+But...
+
+## File changes in the new version of PCRE
+
+First we need to go through what's changed in the new library
+version. Files may have new names, functions may have moved and so on.
+
+Start by building the new library:
+
+      ~/tmp/pcre> cd pcre-8.33/
+      ~/tmp/pcre/pcre-8.33> ./configure --enable-utf --enable-unicode-properties --disable-shared --disable-stack-for-recursion
+      ~/tmp/pcre/pcre-8.33> make
+
+In the make process, you will probably notice most files that are
+used, but you can bet that's not all not all...
+
+To begin with you will need a default table for Latin-1 characters, so:
+
+   	 ~/tmp/pcre/pcre-8.33> cc -DHAVE_CONFIG_H -o dftables dftables.c
+	 ~/tmp/pcre/pcre-8.33> LANG=sv_SE ./dftables -L ../epcre-8.33/pcre_latin_1_table.c
+
+Compare it to the pcre\_latin\_1\_table.c in the old version, they
+should not differ in any significant way.
+
+A good starting point is then to try to find all files in the new
+version of the library that have (probably) the same names as the
+one's in our distribution:
+
+	~/tmp/pcre/pcre-8.33> cd ../epcre-7.6/
+	~/tmp/pcre/epcre-7.6> for x in *.[ch]; do if [ '!' -f ../pcre-8.33/$x ]; then echo $x; else cp ../pcre-8.33/$x ../epcre-8.33/; fi; done
+
+This will output a list of files not found in the new distro. Let's
+look at the list from the example upgrade:
+
+     local_config.h
+     make_latin1_table.c
+     pcre_info.c
+     pcre_latin_1_table.c
+     pcre_make_latin1_default.c
+     pcre_try_flipped.c
+     pcre_ucp_searchfuncs.c
+     ucpinternal.h
+     ucptable.h
+
+* local\_config.h - OK, that's our child, it contains PCRE-specific
+  configure-results (i.e. the #defines that are results from out
+  parameters to configure, like NO\_RECURSE etc). Just copy it and
+  edit it according to what specific settings you can find in the
+  generated config.h from the real library build. In our example case,
+  the #define SUPPORT\_UTF8 should be renamed to #define SUPPORT\_UTF
+  and #define VERSION "7.6" should be changed to #define VERSION
+  "8.33"...
+
+* make\_latin1\_table.c - it was renamed to dftables.c, so we copy
+  that instead.
+
+* pcre\_info.c - It was simply removed from the library. Good, because
+  it was useless... So just ignore.
+
+* pcre\_latin\_1\_table.c - No problem, we generated a new one in the
+  earlier stage.
+
+* pcre\_make\_latin1\_default.c - No longer used, a hack that's not
+  needed with dftables. Ignored
+
+* pcre\_try\_flipped.c - This functionality has been removed from
+  pcre\_exec, you cannot compile on one endianess and execute on
+  another any more :( Ignored.
+
+* pcre\_ucp\_searchfuncs.c, ucpinternal.h, ucptable.h - this
+  functionality is moved to pcre\_ucd.c, copy that one instead.
+
+OK, now go the other way and look at what was actually built for the new version of pcre:
+
+    ~/tmp/pcre/epcre-7.6> cd ../pcre-8.33/
+    ~/tmp/pcre/pcre-8.33> nm ./.libs/libpcre.a | egrep 'lib.*.o:'
+
+The output for this release was:
+
+    libpcre_la-pcre_byte_order.o:
+    libpcre_la-pcre_compile.o:
+    libpcre_la-pcre_config.o:
+    libpcre_la-pcre_dfa_exec.o:
+    libpcre_la-pcre_exec.o:
+    libpcre_la-pcre_fullinfo.o:
+    libpcre_la-pcre_get.o:
+    libpcre_la-pcre_globals.o:
+    libpcre_la-pcre_jit_compile.o:
+    libpcre_la-pcre_maketables.o:
+    libpcre_la-pcre_newline.o:
+    libpcre_la-pcre_ord2utf8.o:
+    libpcre_la-pcre_refcount.o:
+    libpcre_la-pcre_string_utils.o:
+    libpcre_la-pcre_study.o:
+    libpcre_la-pcre_tables.o:
+    libpcre_la-pcre_ucd.o:
+    libpcre_la-pcre_valid_utf8.o:
+    libpcre_la-pcre_version.o:
+    libpcre_la-pcre_xclass.o:
+    libpcre_la-pcre_chartables.o:
+
+Libtool has changed the object names, but we can fix that and see what
+sources we have already decided should exist:
+
+	~/tmp/pcre/pcre-8.33> NAMES=`nm ./.libs/libpcre.a | egrep 'lib.*.o:'| sed 's,libpcre_la-,,' | sed 's,.o:$,,'`
+	~/tmp/pcre/pcre-8.33> for x in $NAMES; do if [ '!' -f ../epcre-8.33/$x.c ]; then echo $x; fi; done
+
+And the list contained:
+
+    pcre_byte_order
+    pcre_jit_compile
+    pcre_string_utils
+
+pcre\_jit\_compile is actually needed, even though we have not enabled
+jit, and the other two contain functionality needed, so just copy the
+sources...
+
+    ~/tmp/pcre/pcre-8.33> for x in $NAMES; do if [ '!' -f ../epcre-8.33/$x.c ]; then cp $x.c ../epcre-8.33/; fi; done
+
+## Test build of stripped down version of new PCRE
+
+Time to do a test build. Copy and edit the pcre.mk makefile and try to
+get something that builds...
+
+I made a wrapper Makefile, hacked pcre.mk a little and did a few
+changes to a few files, namely added:
+  
+	#ifdef ERLANG_INTEGRATION
+	#include "local_config.h"
+	#endif
+
+to pcre\_config.c and pcre\_internal.h. Also pcre.mk needs to get the
+new files added and the old files removed, directory names need to be
+changed and the wrapper can define most. My wrapper Makefile looked
+like this:
+
+    EPCRE_LIB = ./obj/libepcre.a
+    PCRE_GENINC = ./pcre_exec_loop_break_cases.inc
+    PCRE_OBJDIR = ./obj
+    V_AR = ar
+    V_CC = gcc
+    CFLAGS = -g -O2 -DHAVE_CONFIG_H -I/ldisk/pan/git/otp/erts/x86_64-unknown-linux-gnu
+    gen_verbose = 
+    PCRE_DIR=.
+    include pcre.mk
+
+And the according variables were removed together with dependencies
+from pcre.mk. Note that you will need to put things back in order in
+pcre.mk after all testing is done. Once a 'make' is successful, you
+can generate new dependencies:
+
+    ~/tmp/pcre/epcre-8.33> gcc -MM -c -g -O2 -DHAVE_CONFIG_H -I/ldisk/pan/git/otp/erts/x86_64-unknown-linux-gnu -DERLANG_INTEGRATION *.c | grep -v $ERL_TOP
+
+Well, then you have to add $(PCRE\_OBJDIR)/ to each object and
+$(PCRE\_DIR)/ to each header. I did it manually, it's just a couple of
+files. Now your pcre.mk is fairly up to date and it's time to start
+patching in the changes...
+
+## Actually patching in the changes to the C code
+
+### Fixing the functionality (interruptable pcre\_run etc)
+
+Begin with only pcre\_exec.c, that's the important part:
+
+    ~/tmp/pcre/epcre-8.33> cd ../epcre-7.6/
+    ~/tmp/pcre/epcre-7.6> diff -c ../pcre-7.6/pcre_exec.c ./pcre_exec.c > ../epcre_exec.c_7.6.diff
+    ~/tmp/pcre/epcre-7.6> cd ../epcre-8.33 
+
+Now - if you are lucky, you can patch the new pcre\_exec with the
+patch command from the diff, but that may not be the case... Even if:
+
+    ~/tmp/pcre/epcre-8.33> patch -p0 < ../epcre_exec.c_7.6.diff
+
+works like a charm, you still have to go through the main loop and see
+that all do, while and for loops in the code contains COST\_CHK or at
+least COST, or, if it's a small loop (over, say one UTF character),
+mark it as OK with a comment.
+
+You should also check for other changes, like new local variables in
+the pcre\_exec code etc.
+
+What will probably happen, is that the majority of chunks
+fail. pcre\_exec is the main file for PCRE, one that is constantly
+optimized and where every new feature ends up. You will probably see
+so many failed HUNK's that you feel like giving up, but do not
+despair, it's just a matter of patience and hard work:
+
+* First, fix the 'pcre\_exec' function.
+ 
+    * Change the struct PcreExecContext to reflect the local variables
+      in this version of the code.
+
+    * Add/update the defines that makes local variables in the code
+      actually stay in an allocated "exec\_context" and be sure to
+      initialize the "pseudo-stack-variables" in the same way as in
+      the declarations for the original version of the code.
+
+    * The macros SWAPIN and SWAPOUT should be for variables that are
+      used a lot and we do not want to always access through the
+      struct. Also a few parameters are saved by SWAPIN and SWAPOUT.
+
+    * What might be tricky is to get things deallocated in a proper
+      way, there is a function that's called from the BIF code to
+      clean up an exec\_context, be especially observant about how the
+      stack in the 'match' function is allocated! The first frame is
+      supposed to be on the C stack, but in our case is allocated in
+      the exec\_context. The rest of the frames are allocated but
+      never freed, not until the match is done.
+
+      The variable 'frame' in the 'match' function is stored in our
+      additional field of the 'md' structure, that is the stack top,
+      but not necessarily the uppermost frame (due to reuse of old
+      frames, which is supposed to be an optimization...).
+
+    * The housekeeping of the "reduction counter" in the extra\_data
+      struct needs to be added to all places where we break out of the
+      main loop of pcre\_exec. Look for 'break' and you will see the
+      places. Make sure to update
+      '*extra\_data->loop\_counter\_return' whenever you leave this
+      function. It all boils down to some code that loops over the
+      call for match and returns PCRE\_ERROR\_LOOP\_LIMIT and get's
+      jumped back to when the BIF is restarted. You will see it in
+      your diff and you will find a similar place in the new version
+      where you put basically the same code.
+
+    * Fixing pcre\_exec takes about an hour of concentrated work, it
+      could be worse...
+
+* Next, go for the match function. It's simpler in some ways but
+  harder in other. The elimination of the C stack is already there,
+  you just need to modify it a little:
+
+    * In the RRETURN macro for NO\_RECURSE, add updating of
+      md->loop\_limit before returning. You can see how it's done in
+      the diff.
+
+    * RMATCH can be left as it is, at least it could in earlier
+      versions. Note however that you should mimic the allocation
+      strategies of RMATCH and RRETURN in the code at another place
+      later... The principle of the labels HEAP\_RECURSE and
+      HEAP\_RETURN are mimicked by our code in LOOP\_COUNT\_BREAK and
+      LOOP\_COUNT\_RETURN. You'll see later...
+
+    * COST and COST\_CHK, together with the jump to
+      LOOP\_COUNT\_RETURN label are in the beginning of the function
+      'match'. It's a block of macros and declaration of our local
+      variables loop\_count and loop\_limit. We patch in the code for
+      that, but may need to adopt it to new variable names etc. It's
+      important to handle the 'frames' variable correctly, dig it out
+      of the 'md' struct when we are restarting, but initialize it as
+      is done in normal NO\_RECURSE code otherwise. Note that the
+      COST\_CHK macro reuses the Xwhere field of the frame struct, it
+      is not needed when trapping.
+
+    * The LOOP\_COUNT\_BREAK and the LOOP\_COUNT\_RETURN code can now
+      be added. Make sure to check both how a new stack frame should be
+      properly allocated by mimicking the code in RMATCH, and how (if)
+      it should be freed by mimicking RRETURN. Also check which
+      variables need to be saved. They are properly pointed out in
+      8.33 with the comment 'These variables do not need to be
+      preserved over recursion' and appear in the beginning of the
+      function. Find variables of similar type in the frame structure
+      and reuse them. In 8.33 there are eight such variables. They are
+      placed at the end of the function 'match'. If You are reading
+      the diff, you need to scroll past all the COST\_CHK calls,
+      i.e. past the whole regexp machine loop.
+
+    * Now take the time to add things like debug macros to the top of
+      the file and one single COST\_CHK (preferably the one right
+      after for(;;) in 'match'), and see if you can compile. You will
+      probably need to add some fields in the structures in pcre.h,
+      see from a larger diff what you need there and iterate until you
+      can compile.
+
+    * So, what's left is to add all the COST and COST\_CHK macros,
+      plus marking all harmless loops as OK. There are a few rules
+      here:
+
+        * Mark *every* loop with the comment 'LOOP\_COUNT: xxxx',
+          where xxxx is either 'Ok', 'COST' or 'CHK'. There are 175
+          'LOOP\_COUNT:' comments in 8.33.
+
+        * Loops marked 'Ok' need no macro, either because they are so
+          short (like over an UTF character) or because they contain
+          an RMATCH macro, in which case they will be accounted for
+          anyway.
+
+        * Loops marked 'COST' will have an associated 'COST(N)' macro,
+          either before, if we know the amount of iterations, or
+          within. Reductions are counted, but we will not
+          interrupt. This is typically in what is expected to be
+          medium long loops or at places where interruption is hard
+          (like where we have local variables that are alive. The
+          selection between 'COST' and 'COST\_CHK' is hard. 'COST' is
+          much cheaper and usually enough, but when in doubt about the
+          loop length, try to use 'COST\_CHK', while making very sure
+          there are no live block-local variables that need to be
+          saved over the trap. There are 49 'COST' macros in 8.33.
+
+        * Loops marked 'CHK' shall contain a 'COST\_CHK(N)'
+          macro. This macro both counts reductions and may result in
+          an interrupt and a return to Erlang space. It is expensive
+          and it is vital to ensure that there are no unexpected local
+          variables that live past the macro. Most variables are in
+          the pseudo stack frame, but some regexp instructions declare
+          temporaries inside blocks. Make sure they are not expected
+          to be alive after a COST\_CHK if they are not in the
+          'heapframe' structure. If they are, you need to
+          conditionally move them to the 'heapframe' #if
+          defined(ERLANG\_INTEGRATION). in 8.33 the variables 'lgb'
+          and 'rgb' are preserved in this way. There are 54
+          'COST\_CHK's in 8.33.
+
+        * I've marked a few block-local variables with warnings, but
+          look thoroughly through the main loop to detect any new
+          ones.
+
+        * Be careful when it comes to freeing the context from Erlang
+          (the function erts\_pcre\_free\_restart\_data), Whatever is
+          done there has to work *both* when the context is freed in
+          the middle of an operation (because of trapping) and when
+          some things have been freed by a successful
+          return. Specifically, make sure to set md->offset\_vector to
+          NULL whenever it's freed (in the rest of the code) and
+          construct release\_match\_heapframes so that it can be
+          called multiple times for the same heapframe (set the next
+          pointer in the "static" frame, i.e. the one allocated in the
+          md to NULL after freeing).
+
+    * To add the costs to the main loop takes less than one work day,
+      keep calm and continue...
+
+OK, now you are done with the pcre\_exec (or at least, you think
+so). The rest is simpler. You have probably already handled 'pcre.h'
+and 'pcre\_internal.h' to add fields to the structures etc. Looking at
+a diff from an earlier version, you will see what's left. In upgrading
+to 8.33, the following things was left to do after pcre\_exec was
+fixed, remember you could generate a diff with:
+
+    ~/tmp/pcre/epcre-8.33> cd ../epcre-7.6/
+    ~/tmp/pcre/epcre-7.6> (for x in *.[ch]; do if [ -f ../pcre-7.6/$x ]; then diff -c ../pcre-7.6/$x $x; fi; done) > ../epcre-7.6.diff 
+
+Open the diff in your favorite editor and remove whatever changes you
+have already made, like everything that has to do with pcre\_exec.c
+and probably a large part of pcre.h/pcre\_internal.h.
+
+The expected result is a diff that either contains only the
+'%ExternalCopyright%' comments or contains them and the addition of
+the erts\_ prefix, depending on if you reverted the prefix change
+(using 'git revert') before starting to work. With a little luck, the
+patch of the remaining stuff should be possible to apply
+automatically. If anything fails, just add it manually.
+
+### Fixing the erts\_prefix
+
+The erts\_ prefix is mostly implemented by adding '#if
+defined(ERLANG\_INTEGRATION)' to a lot of function headers, inside the
+COMPILE\_UTF8 part. If you then also change the PRIV and PUBL macros
+in pcre\_internal.h. Typical diffs look like:
+
+        #if defined COMPILE_PCRE8
+      + #if defined(ERLANG_INTEGRATION)
+      + #ifndef PUBL
+      + #define PUBL(name) erts_pcre_##name
+      + #endif
+      + #ifndef PRIV
+      + #define PRIV(name) _erts_pcre_##name
+      + #endif
+      + #else
+        #ifndef PUBL
+        #define PUBL(name) pcre_##name
+        #endif
+        #ifndef PRIV
+        #define PRIV(name) _pcre_##name
+        #endif
+      + #endif
+
+and
+
+	    #if defined COMPILE_PCRE8
+	  + #if defined(ERLANG_INTEGRATION)
+	  + PCRE_EXP_DECL int erts_pcre_pattern_to_host_byte_order(pcre *argument_re,
+	  +   erts_pcre_extra *extra_data, const unsigned char *tables)
+	  + #else
+  	    PCRE_EXP_DECL int pcre_pattern_to_host_byte_order(pcre *argument_re,
+    	      pcre_extra *extra_data, const unsigned char *tables)
+	  + #endif
+
+Note that some data types, like pcre\_extra are accessed with the PUBL
+macro, so they need to explicitly get the prefix added. pcre.h is a
+pig, as it declares prototypes for all functions regardless of
+compilation ode, so there is quite a lot of '#if
+defined(ERLANG\_INTEGRATION)' to add there.
+
+Anyway, now try to patch, using a diff where you have removed the
+changes you made manually (probably to pcre\_exec.c) but make sure to
+save your work (temporary git repository?) before, so you can revert
+any disasters...
+
+    ~/tmp/pcre/epcre-7.6> cd ../epcre-8.33/
+    ~/tmp/pcre/epcre-8.33> patch -p0 < ../epcre-7.6_clean2.diff
+
+Some hunks may certainly still fail, read through the .rej file and fix it.
+
+### ExternalCopyright
+
+Now you should check that the 'ExternalCopyright' comment is present
+in all source files:
+
+    ~/tmp/pcre/epcre-8.33> for x in *.[ch]; do if grep ExternalCopyright $x > /dev/null; then true; else echo $x; fi; done
+
+In this upgrade (from 7.6 to 8.33) we certainly had some new and
+renamed files:
+
+    dftables.c
+    pcre_byte_order.c
+    pcre_chartables.c
+    pcre_jit_compile.c
+    pcre_latin_1_table.c
+    pcre_string_utils.c
+    pcre_ucd.c
+
+Go through them manually and add the 'ExternalCopyright' comment.
+
+## Integrate with Erlang
+
+Now you are done with most of the tedious work. It's time to move this
+into your branch of the Erlang source tree, remove old files and add
+new ones, plus add the tar file with the original pcre dist. Remember
+to fix your hacked version of pcre.mk and then try to build
+Erlang. You might need to update 'erl\_bif\_re.c' to reflect any
+changes in the PCRE library. When it builds, run the test suites.
+
+Make sure to rename any files that has new names and remove any files
+that are no longer present before copying in the new versions from
+your temporary directory. In our example we remove 'pcre\_info.c',
+'pcre\_make\_latin1\_default.c', 'pcre\_try\_flipped.c',
+'ucpinternal.h' and 'ucptable.h'. We rename 'make\_latin1\_table.c' to
+'dftables.c' and 'pcre\_ucp\_searchfuncs.c' to 'pcre\_ucd.c'.
+
+After copying in the sources, we can try to build. Do not forget to
+fix whatever you did in pcre.mk to make it build locally.
+
+## Update test suites
+
+The next step is to integrate the updated PCRE tests into our test suites.
+
+Copy testoutput[1-9] from the testdata directory of your new version
+of pcre, to the re\_SUITE\_data in stdlib's test suites. Run the
+test suites and remove any bugs. Usually the bugs come from the fact
+that the PCRE test suites get better and from our implementation of
+global matching, which may have bugs outside of the PCRE library. The
+test suite 'pcre' is the one that runs these tests. Also copy
+testoutput11-8 to testoutput10, the testoutput10 file in pcre is
+nowadays for the DFA, which we do not use.
+
+The next step is to regenerate re\_testoutput1\_replacement\_test. How
+to do that is in a comment in the beginning of the file. The key
+module is run\_pcre\_tests.erl, which both driver the pcre test and
+generate re\_testoutput1\_replacement\_test.erl. Watch during the
+generation that you do not get to many of the "Fishy character"
+messages, if they are more than, say 20, you will probably need to
+address the UTF8 issues in the Perl execution. As it is now, we skip
+non latin1 characters in this test. You will need to run iconv on the
+generated module to make it UTF-8 before running tests.
+
+The exact same procedure goes for the re\_testoutput1\_split\_test.erl. 
+
+Also add copyright headers to the files after converting them to UTF-8.
+
+After ironing out the rest of the bugs, you should be done with the
+code.
+
+## Update documentation
+
+Now it's time for the documentation, which is fairly
+straightforward. Diff the pcrepattern man pages from the old and new
+PCRE distros and update the re.xml file accordingly. It may help to
+have the generated HTML file from the new version to cut and paste
+from, but as you will notice, it's quite a few changes from HTML to
+XML. All lists are reformatted, the &lt;pre&gt; tags are made into
+either &lt;code&gt; or &lt;quite&gt; etc. Also the &lt;P&gt; tags are
+converted to lowercase and all mentioned options and function calls
+are converted to their Erlang counterpart. Really awesome work that
+requires thorough reading of all new text.  For the upgrade from 7.6
+to 8.33, the update of the pcrepattern part of our manual page took
+about eight hours.
+
+## Add new relevant options to re
+
+Then, when all this is done, you should add any new relevant options
+from the PCRE library to both the code (erl\_bif\_re.c), the specs and
+the Erlang function 'copt/1' (re.erl) and the manual page
+(re.xml). Make sure the options are really relevant to add to the
+Erlang API, check if they are compile or run-time options (or both) and
+add them to the 'parse\_options' function of erl\_bif\_re.c. Adding an
+option that is just passed through to PCRE is pretty simple, at least
+"code wise".
+ 
+Now you are done. Run all test suites on all machines and you will be happy.
+
+## Final notes
+
+To avoid the work of a major upgrade, it is probably worth it to keep
+in pace with the changes to PCRE. The upgrade from 7.6 to 8.33,
+including tracking down bugs etc, took me a total of two weeks. If
+smaller diffs from the PCRE development were integrated in a more
+incremental fashion, it will be much easier each time and you will
+have the PCRE library up to date. PCRE should probably be updated for
+each major release, instead of every five years...
\ No newline at end of file