guestfs-hacking(1)
NAME
guestfs-hacking - extending and contributing to libguestfs
DESCRIPTION
This manual page is for hackers who want to extend libguestfs itself.
OVERVIEW OF THE SOURCE CODE
Libguestfs source is located in the github repository
https://github.com/libguestfs/libguestfs
Large amounts of boilerplate code in libguestfs (RPC, bindings,
documentation) are generated. This means that many source files will
appear to be missing from a straightforward git checkout. You have to
run the generator ("./autogen.sh && make -C generator") in order to
create those files.
Libguestfs uses an autotools-based build system, with the main files
being configure.ac and Makefile.am. The generator subdirectory
contains the generator, plus files describing the API. The src
subdirectory contains source for the library. The appliance and daemon
subdirectories contain the source for the code that builds the
appliance, and the code that runs in the appliance respectively. Other
directories are covered in the section "SOURCE CODE SUBDIRECTORIES"
below.
Apart from the fact that all API entry points go via some generated
code, the library is straightforward. (In fact, even the generated
code is designed to be readable, and should be read as ordinary code).
Some actions run entirely in the library, and are written as C
functions in files under src. Others are forwarded to the daemon where
(after some generated RPC marshalling) they appear as C functions in
files under daemon.
To build from source, first read the "README" file.
local* FILES
Files in the top source directory that begin with the prefix local* are
ignored by git. These files can contain local configuration or scripts
that you need to build libguestfs.
By convention, I have a file called localconfigure which is a simple
wrapper around autogen.sh containing local configure customizations
that I need:
. localenv
./autogen.sh \
--with-default-backend=libvirt \
--enable-gcc-warnings \
--enable-gtk-doc \
-C \
"$@"
So I can use this to build libguestfs:
./localconfigure && make
If there is a file in the top build directory called localenv, then it
will be sourced by "make". This file can contain any local environment
variables needed, eg. for skipping tests:
# Use an alternate python binary.
export PYTHON=python3
# Skip this test, it is broken.
export SKIP_TEST_BTRFS_FSCK=1
Note that localenv is included by the top Makefile (so it's a Makefile
fragment). But if it is also sourced by your localconfigure script
then it is used as a shell script.
ADDING A NEW API ACTION
Because large amounts of boilerplate code in libguestfs are generated,
this makes it easy to extend the libguestfs API.
To add a new API action there are two changes:
1. You need to add a description of the call (name, parameters, return
type, tests, documentation) to generator/actions.ml.
There are two sorts of API action, depending on whether the call
goes through to the daemon in the appliance, or is serviced
entirely by the library (see "ARCHITECTURE" in
guestfs-internals(3)). "guestfs_sync" in guestfs(3) is an example
of the former, since the sync is done in the appliance.
"guestfs_set_trace" in guestfs(3) is an example of the latter,
since a trace flag is maintained in the handle and all tracing is
done on the library side.
Most new actions are of the first type, and get added to the
"daemon_functions" list. Each function has a unique procedure
number used in the RPC protocol which is assigned to that action
when we publish libguestfs and cannot be reused. Take the latest
procedure number and increment it.
For library-only actions of the second type, add to the
"non_daemon_functions" list. Since these functions are serviced by
the library and do not travel over the RPC mechanism to the daemon,
these functions do not need a procedure number, and so the
procedure number is set to "-1".
2. Implement the action (in C):
For daemon actions, implement the function "do_<name>" in the
"daemon/" directory.
For library actions, implement the function "guestfs_impl_<name>"
in the "src/" directory.
In either case, use another function as an example of what to do.
After making these changes, use "make" to compile.
Note that you don't need to implement the RPC, language bindings,
manual pages or anything else. It's all automatically generated from
the OCaml description.
ADDING TESTS FOR AN API ACTION
You can supply zero or as many tests as you want per API call. The
tests can either be added as part of the API description
(generator/actions.ml), or in some rarer cases you may want to drop a
script into "tests/*/". Note that adding a script to "tests/*/" is
slower, so if possible use the first method.
The following describes the test environment used when you add an API
test in actions.ml.
The test environment has 4 block devices:
/dev/sda 2 GB
General block device for testing.
/dev/sdb 2 GB
/dev/sdb1 is an ext2 filesystem used for testing filesystem write
operations.
/dev/sdc 10 MB
Used in a few tests where two block devices are needed.
/dev/sdd
ISO with fixed content (see images/test.iso).
To be able to run the tests in a reasonable amount of time, the
libguestfs appliance and block devices are reused between tests. So
don't try testing "guestfs_kill_subprocess" in guestfs(3) :-x
Each test starts with an initial scenario, selected using one of the
"Init*" expressions, described in generator/types.ml. These initialize
the disks mentioned above in a particular way as documented in
types.ml. You should not assume anything about the previous contents
of other disks that are not initialized.
You can add a prerequisite clause to any individual test. This is a
run-time check, which, if it fails, causes the test to be skipped.
Useful if testing a command which might not work on all variations of
libguestfs builds. A test that has prerequisite of "Always" means to
run unconditionally.
In addition, packagers can skip individual tests by setting environment
variables before running "make check".
SKIP_TEST_<CMD>_<NUM>=1
eg: "SKIP_TEST_COMMAND_3=1" skips test #3 of "guestfs_command" in
guestfs(3).
or:
SKIP_TEST_<CMD>=1
eg: "SKIP_TEST_ZEROFREE=1" skips all "guestfs_zerofree" in guestfs(3)
tests.
Packagers can run only certain tests by setting for example:
TEST_ONLY="vfs_type zerofree"
See tests/c-api/tests.c for more details of how these environment
variables work.
DEBUGGING NEW API ACTIONS
Test new actions work before submitting them.
You can use guestfish to try out new commands.
Debugging the daemon is a problem because it runs inside a minimal
environment. However you can fprintf messages in the daemon to stderr,
and they will show up if you use "guestfish -v".
ADDING A NEW LANGUAGE BINDING
All language bindings must be generated by the generator (see the
generator subdirectory).
There is no documentation for this yet. We suggest you look at an
existing binding, eg. generator/ocaml.ml or generator/perl.ml.
ADDING TESTS FOR LANGUAGE BINDINGS
Language bindings should come with tests. Previously testing of
language bindings was rather ad-hoc, but we have been trying to
formalize the set of tests that every language binding should use.
Currently only the OCaml and Perl bindings actually implement the full
set of tests, and the OCaml bindings are canonical, so you should
emulate what the OCaml tests do.
This is the numbering scheme used by the tests:
- 000+ basic tests:
010 load the library
020 create
030 create-flags
040 create multiple handles
050 test setting and getting config properties
060 explicit close
065 implicit close (in GC'd languages)
070 optargs
080 version
- 100 launch, create partitions and LVs and filesystems
- 400+ events:
410 close event
420 log messages
430 progress messages
- 800+ regression tests (specific to the language)
- 900+ any other custom tests for the language
To save time when running the tests, only 100, 430, 800+, 900+ should
launch the handle.
FORMATTING CODE
Our C source code generally adheres to some basic code-formatting
conventions. The existing code base is not totally consistent on this
front, but we do prefer that contributed code be formatted similarly.
In short, use spaces-not-TABs for indentation, use 2 spaces for each
indentation level, and other than that, follow the K&R style.
If you use Emacs, add the following to one of one of your start-up
files (e.g., ~/.emacs), to help ensure that you get indentation right:
;;; In libguestfs, indent with spaces everywhere (not TABs).
;;; Exceptions: Makefile and ChangeLog modes.
(add-hook 'find-file-hook
'(lambda () (if (and buffer-file-name
(string-match "/libguestfs\\>"
(buffer-file-name))
(not (string-equal mode-name "Change Log"))
(not (string-equal mode-name "Makefile")))
(setq indent-tabs-mode nil))))
;;; When editing C sources in libguestfs, use this style.
(defun libguestfs-c-mode ()
"C mode with adjusted defaults for use with libguestfs."
(interactive)
(c-set-style "K&R")
(setq c-indent-level 2)
(setq c-basic-offset 2))
(add-hook 'c-mode-hook
'(lambda () (if (string-match "/libguestfs\\>"
(buffer-file-name))
(libguestfs-c-mode))))
TESTING YOUR CHANGES
Turn warnings into errors when developing to make warnings hard to
ignore:
./configure --enable-werror
Useful targets are:
"make check"
Runs the regular test suite.
This is implemented using the regular automake "TESTS" target. See
the automake documentation for details.
"make check-valgrind"
Runs a subset of the test suite under valgrind.
See "VALGRIND" below.
"make check-valgrind-local-guests"
Runs a subset of the test suite under valgrind using locally
installed libvirt guests (read-only).
"make check-direct"
Runs all tests using default appliance back-end. This only has any
effect if a non-default backend was selected using "./configure
--with-default-backend=..."
"make check-valgrind-direct"
Run a subset of the test suite under valgrind using the default
appliance back-end.
"make check-uml"
Runs all tests using the User-Mode Linux backend.
As there is no standard location for the User-Mode Linux kernel,
you have to set "LIBGUESTFS_HV" to point to the kernel image, eg:
make check-uml LIBGUESTFS_HV=~/d/linux-um/vmlinux
"make check-valgrind-uml"
Runs all tests using the User-Mode Linux backend, under valgrind.
As above, you have to set "LIBGUESTFS_HV" to point to the kernel.
"make check-with-upstream-qemu"
Runs all tests using a local qemu binary. It looks for the qemu
binary in QEMUDIR (defaults to $HOME/d/qemu), but you can set this
to another directory on the command line, eg:
make check-with-upstream-qemu QEMUDIR=/usr/src/qemu
"make check-with-upstream-libvirt"
Runs all tests using a local libvirt. This only has any effect if
the libvirt backend was selected using "./configure
--with-default-backend=libvirt"
It looks for libvirt in LIBVIRTDIR (defaults to $HOME/d/libvirt),
but you can set this to another directory on the command line, eg:
make check-with-upstream-libvirt LIBVIRTDIR=/usr/src/libvirt
"make check-slow"
Runs some slow/long-running tests which are not run by default.
To mark a test as slow/long-running:
* Add it to the list of "TESTS" in the Makefile.am, just like a
normal test.
* Modify the test so it checks if the "SLOW=1" environment
variable is set, and if not set it skips (ie. returns with exit
code 77).
* Add a variable "SLOW_TESTS" to the Makefile.am listing the slow
tests.
* Add a rule to the Makefile.am:
check-slow:
$(MAKE) check TESTS="$(SLOW_TESTS)" SLOW=1
"make check-all"
Equivalent to running all "make check*" rules.
"make check-release"
Runs a subset of "make check*" rules that are required to pass
before a tarball can be released. Currently this is:
* check
* check-valgrind
* check-direct
* check-valgrind-direct
* check-slow
"make installcheck"
Run "make check" on the installed copy of libguestfs.
The version of installed libguestfs being tested, and the version
of the libguestfs source tree must be the same.
Do:
./autogen.sh
make clean ||:
make
make installcheck
VALGRIND
When you do "make check-valgrind", it searches for any Makefile.am in
the tree that has a "check-valgrind:" target and runs it.
Writing the Makefile.am and tests correctly to use valgrind and working
with automake parallel tests is subtle.
If your tests are run via a shell script wrapper, then in the wrapper
use:
$VG virt-foo
and in the Makefile.am use:
check-valgrind:
make VG="$(top_builddir)/run @VG@" check
However, if your binaries run directly from the "TESTS" rule, you have
to modify the Makefile.am like this:
LOG_COMPILER = $(VG)
check-valgrind:
make VG="@VG@" check
In either case, check that the right program is being tested by
examining the tmp/valgrind* log files carefully.
DAEMON CUSTOM PRINTF FORMATTERS
In the daemon code we have created custom printf formatters %Q and %R,
which are used to do shell quoting.
%Q Simple shell quoted string. Any spaces or other shell characters
are escaped for you.
%R Same as %Q except the string is treated as a path which is prefixed
by the sysroot.
For example:
asprintf (&cmd, "cat %R", path);
would produce "cat /sysroot/some\ path\ with\ spaces"
Note: Do not use these when you are passing parameters to the
"command{,r,v,rv}()" functions. These parameters do NOT need to be
quoted because they are not passed via the shell (instead, straight to
exec). You probably want to use the "sysroot_path()" function however.
SUBMITTING YOUR NEW API ACTIONS
Submit patches to the mailing list:
http://www.redhat.com/mailman/listinfo/libguestfs and CC to
[email protected].
INTERNATIONALIZATION (I18N) SUPPORT
We support i18n (gettext anyhow) in the library.
However many messages come from the daemon, and we don't translate
those at the moment. One reason is that the appliance generally has
all locale files removed from it, because they take up a lot of space.
So we'd have to readd some of those, as well as copying our PO files
into the appliance.
Debugging messages are never translated, since they are intended for
the programmers.
SOURCE CODE SUBDIRECTORIES
align
virt-alignment-scan(1) command and documentation.
appliance
The libguestfs appliance, build scripts and so on.
bash
Bash tab-completion scripts.
build-aux
Various build scripts used by autotools.
builder
virt-builder(1) command and documentation.
cat The virt-cat(1), virt-filesystems(1), virt-log(1) and virt-ls(1)
commands and documentation.
contrib
Outside contributions, experimental parts.
customize
virt-customize(1) command and documentation.
daemon
The daemon that runs inside the libguestfs appliance and carries
out actions.
df virt-df(1) command and documentation.
dib virt-dib(1) command and documentation.
diff
virt-diff(1) command and documentation.
docs
Miscellaneous manual pages.
edit
virt-edit(1) command and documentation.
examples
C API example code.
fish
guestfish(1), the command-line shell, and various shell scripts
built on top such as virt-copy-in(1), virt-copy-out(1),
virt-tar-in(1), virt-tar-out(1).
format
virt-format(1) command and documentation.
fuse
guestmount(1), FUSE (userspace filesystem) built on top of
libguestfs.
generator
The crucially important generator, used to automatically generate
large amounts of boilerplate C code for things like RPC and
bindings.
get-kernel
virt-get-kernel(1) command and documentation.
gnulib
Gnulib is used as a portability library. A copy of gnulib is
included under here.
inspector
virt-inspector(1), the virtual machine image inspector.
logo
Logo used on the website. The fish is called Arthur by the way.
m4 M4 macros used by autoconf.
make-fs
virt-make-fs(1) command and documentation.
mllib
Various libraries and common code used by virt-resize(1) and the
other tools which are written in OCaml.
p2v virt-p2v(1) command, documentation and scripts for building the
virt-p2v ISO or disk image.
po Translations of simple gettext strings.
po-docs
The build infrastructure and PO files for translations of manpages
and POD files. Eventually this will be combined with the po
directory, but that is rather complicated.
rescue
virt-rescue(1) command and documentation.
resize
virt-resize(1) command and documentation.
sparsify
virt-sparsify(1) command and documentation.
src Source code to the C library.
sysprep
virt-sysprep(1) command and documentation.
tests
Tests.
test-data
Files and other test data used by the tests.
test-tool
Test tool for end users to test if their qemu/kernel combination
will work with libguestfs.
tmp Used for temporary files when running the tests (instead of /tmp
etc). The reason is so that you can run multiple parallel tests of
libguestfs without having one set of tests overwriting the
appliance created by another.
tools
Command line tools written in Perl (virt-win-reg(1) and many
others).
v2v virt-v2v(1) command and documentation.
website
The http://libguestfs.org website files.
csharp
erlang
gobject
golang
haskell
java
lua
ocaml
php
perl
python
ruby
Language bindings.
MAKING A STABLE RELEASE
When we make a stable release, there are several steps documented here.
See "LIBGUESTFS VERSION NUMBERS" in guestfs(3) for general information
about the stable branch policy.
* Check "make && make check" works on at least Fedora, Debian and
Ubuntu.
* Check "./configure --without-libvirt" works.
* Finalize guestfs-release-notes.pod
* Push and pull from Zanata.
Run:
zanata push
to push the latest POT files to Zanata. Then run:
./zanata-pull.sh
which is a wrapper to pull the latest translated *.po files.
* Consider updating gnulib to latest upstream version.
* Create new stable and development directories under
http://libguestfs.org/download.
* Edit website/index.html.in.
* Set the version (in configure.ac) to the new stable version, ie.
1.XX.0, and commit it:
./localconfigure
make distclean -k
./localconfigure
make && make dist
make maintainer-commit
make maintainer-tag
* Create the stable branch in git:
git branch stable-1.XX
git push origin stable-1.XX
* Do a full release of the stable branch.
* Set the version to the next development version and commit that.
Optionally do a full release of the development branch.
SEE ALSO
guestfs(3), guestfs-examples(3), guestfs-internals(3), guestfs-performance(1), guestfs-release-notes(1), guestfs-testing(1), libguestfs-test-tool(1), libguestfs-make-fixed-appliance(1), http://libguestfs.org/.
AUTHORS
Richard W.M. Jones ("rjones at redhat dot com")
COPYRIGHT
Copyright (C) 2009-2016 Red Hat Inc.
LICENSE
This library is free software; you can redistribute it and/or modify it under the terms of the GNU Lesser General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version. This library is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Lesser General Public License for more details. You should have received a copy of the GNU Lesser General Public License along with this library; if not, write to the Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
BUGS
To get a list of bugs against libguestfs, use this link:
https://bugzilla.redhat.com/buglist.cgi?component=libguestfs&product=Virtualization+Tools
To report a new bug against libguestfs, use this link:
https://bugzilla.redhat.com/enter_bug.cgi?component=libguestfs&product=Virtualization+Tools
When reporting a bug, please supply:
* The version of libguestfs.
* Where you got libguestfs (eg. which Linux distro, compiled from
source, etc)
* Describe the bug accurately and give a way to reproduce it.
* Run libguestfs-test-tool(1) and paste the complete, unedited output
into the bug report.
Opportunity
Personal Opportunity - Free software gives you access to billions of dollars of software at no cost. Use this software for your business, personal use or to develop a profitable skill. Access to source code provides access to a level of capabilities/information that companies protect though copyrights. Open source is a core component of the Internet and it is available to you. Leverage the billions of dollars in resources and capabilities to build a career, establish a business or change the world. The potential is endless for those who understand the opportunity.
Business Opportunity - Goldman Sachs, IBM and countless large corporations are leveraging open source to reduce costs, develop products and increase their bottom lines. Learn what these companies know about open source and how open source can give you the advantage.
Free Software
Free Software provides computer programs and capabilities at no cost but more importantly, it provides the freedom to run, edit, contribute to, and share the software. The importance of free software is a matter of access, not price. Software at no cost is a benefit but ownership rights to the software and source code is far more significant.
Free Office Software - The Libre Office suite provides top desktop productivity tools for free. This includes, a word processor, spreadsheet, presentation engine, drawing and flowcharting, database and math applications. Libre Office is available for Linux or Windows.
Free Books
The Free Books Library is a collection of thousands of the most popular public domain books in an online readable format. The collection includes great classical literature and more recent works where the U.S. copyright has expired. These books are yours to read and use without restrictions.
Source Code - Want to change a program or know how it works? Open Source provides the source code for its programs so that anyone can use, modify or learn how to write those programs themselves. Visit the GNU source code repositories to download the source.
Education
Study at Harvard, Stanford or MIT - Open edX provides free online courses from Harvard, MIT, Columbia, UC Berkeley and other top Universities. Hundreds of courses for almost all major subjects and course levels. Open edx also offers some paid courses and selected certifications.
Linux Manual Pages - A man or manual page is a form of software documentation found on Linux/Unix operating systems. Topics covered include computer programs (including library and system calls), formal standards and conventions, and even abstract concepts.
