Support for HTTP(S) Proxy (#655 )

Works like with standard Go client
Update gopacket
2026-10-19 16:15:42 +00:00 · 2019-02-23 10:46:53 +01:00 · 2019-02-16 10:13:03 +01:00 · 2019-02-16 09:55:34 +01:00 · 2019-02-16 09:51:54 +01:00 · 2019-02-16 09:49:29 +01:00
420 changed files with 149928 additions and 580 deletions
@@ -36,7 +36,7 @@ under the terms of the GNU Lesser General Public License versions 3.0

 5. Fees and Payment. The Software license fees will be due and payable in full as set forth in the applicable invoice or at the time of purchase. If the Software does not function properly within two weeks of purchase, please contact us within those two weeks for a refund. You shall be responsible for all taxes, withholdings, duties and levies arising from the order (excluding taxes based on the net income of GoReplay LLC). 

-6. Support, Maintenance and Services. Subject to the terms and conditions of this Agreement, as set forth in your invoice, and as set forth on the GoReplay Pro support page (https://github.com/buger/gor/wiki/Pro-Support), support and maintenance services may be included with the purchase of your license subscription.
+6. Support, Maintenance and Services. Subject to the terms and conditions of this Agreement, as set forth in your invoice, and as set forth on the GoReplay Pro support page (https://github.com/buger/goreplay/wiki/Pro-Support), support and maintenance services may be included with the purchase of your license subscription.

 7. Term of Agreement. 

@@ -48,10 +48,11 @@ under the terms of the GNU Lesser General Public License versions 3.0

 8. Disclaimer of Warranties. The Software is provided "as is," with all faults, defects and errors, and without warranty of any kind. GoReplay LLC does not warrant that the Software will be free of bugs, errors, viruses or other defects, and GoReplay LLC shall have no liability of any kind for the use of or inability to use the Software, the Software content or any associated service, and you acknowledge that it is not technically practicable for GoReplay LLC to do so. 
 To the maximum extent permitted by applicable law, GoReplay LLC disclaims all warranties, express, implied, arising by law or otherwise, regarding the Software, the Software content and their respective performance or suitability for your intended use, including without limitation any implied warranty of merchantability, fitness for a particular purpose.
+Notwithstanding the foregoing, GoReplay LLC represents and warrants that it either owns the entire right to, title to, interest in, or has the right to license, the Software, and your proper use of the same will not violate any intellectual property or proprietary rights of another.

 9. Limitation of Liability. 

-In no event will GoReplay LLC be liable for any direct, indirect, consequential, incidental, special, exemplary, or punitive damages or liabilities whatsoever arising from or relating to the Software, the Software content or this Agreement, whether based on contract, tort (including negligence), strict liability or other theory, even if GoReplay LLC has been advised of the possibility of such damages.
+In no event will either party be liable for any direct, indirect, consequential, incidental, special, exemplary, or punitive damages or liabilities whatsoever arising from or relating to the Software, the Software content or this Agreement, whether based on contract, tort (including negligence), strict liability or other theory, even if such party has been advised of the possibility of such damages.

 In no event will GoReplay LLC liability exceed the Software license price as indicated in the invoice. The existence of more than one claim will not enlarge or extend this limit.

@@ -59,9 +60,9 @@ In no event will GoReplay LLC liability exceed the Software license price as ind

 11. Acknowledgements.

-11.1 Consent to the Use of Data. You agree that GoReplay LLC and its affiliates may collect and use technical information gathered as part of the product support services. GoReplay LLC may use this information solely to improve products and services and will not disclose this information in a form that personally identifies you.
+11.1 Consent to the Use of Data. You agree that GoReplay LLC and its affiliates may collect and use technical information gathered as part of the product support services to be used in an anonymized manner. GoReplay LLC may use this information solely to improve products and services and will not disclose this information in a form that personally identifies you.

-11.2 Verification. We or a certified auditor acting on our behalf, may, upon its reasonable request and at its expense, audit you with respect to the use of the Software. Such audit may be conducted by mail, electronic means or through an in-person visit to your place of business. Any such in-person audit shall be conducted during regular business hours at your facilities and shall not unreasonably interfere with your business activities. We shall not remove, copy, or redistribute any electronic material during the course of an audit. If an audit reveals that you are using the Software in a way that is in material violation of the terms of the EULA, then you shall pay our reasonable costs of conducting the audit. In the case of a material violation, you agree to pay Us any amounts owing that are attributable to the unauthorized use. In the alternative, We reserve the right, at our sole option, to terminate the licenses for the Software.
+11.2 Verification. We or a certified auditor acting on our behalf, may, upon its reasonable request and at its expense, audit you with respect to the use of the Software, subject to your reasonable information security and confidentiality protections. Such audit may be conducted by mail, electronic means or through an in-person visit to your place of business. Any such in-person audit shall be conducted during regular business hours at your facilities and shall not unreasonably interfere with your business activities. We shall not remove, copy, or redistribute any electronic material during the course of an audit. If an audit reveals that you are using the Software in a way that is in material violation of the terms of the EULA, then you shall pay our reasonable costs of conducting the audit. In the case of a material violation, you agree to pay Us any amounts owing that are attributable to the unauthorized use. In the alternative, We reserve the right, at our sole option, to terminate the licenses for the Software.

 11.3 Government End Users. If the Software and related documentation are supplied to or purchased by or on behalf of the United States Government, then the Software is deemed to be "commercial software" as that term is used in the Federal Acquisition Regulation system. Rights of the United States shall not exceed the minimum rights set forth in FAR 52.227-19 for "restricted computer software". All other terms and conditions of this Agreement apply.

@@ -72,11 +73,11 @@ In no event will GoReplay LLC liability exceed the Software license price as ind

 13.1 Entire Agreement. This Agreement sets forth our entire agreement with respect to the Software and the subject matter hereof and supersedes all prior and contemporaneous understandings and agreements whether written or oral.

-13.2 Amendment. GoReplay LLC reserves the right, in its sole discretion, to amend this Agreement from time. Amendments to this Agreement can be located at: https://github.com/buger/gor/blob/master/COMM-LICENSE.
+13.2 Amendment. GoReplay LLC reserves the right, in its sole discretion, to amend this Agreement from time. Amendments to this Agreement can be located at: https://github.com/buger/goreplay/blob/master/COMM-LICENSE.

 13.3 Assignment. You may not assign this Agreement or any of its rights under this Agreement without the prior written consent of GoReplay LLC and any attempted assignment without such consent shall be void.

-13.4 Export Compliance. You agree to comply with all applicable laws and regulations, including laws, regulations, orders or other restrictions on export, re-export or redistribution of software.
+13.4 Export Compliance. Each party agrees to comply with all applicable laws and regulations, including laws, regulations, orders or other restrictions on export, re-export or redistribution of software.

 13.5 Indemnification. You agree to defend, indemnify, and hold harmless GoReplay LLC from and against any lawsuits, claims, losses, damages, fines and expenses (including attorneys' fees and costs) arising out of your use of the Software or breach of this Agreement.

@@ -1,24 +1,8 @@
-FROM golang:latest
+FROM alpine:latest as builder
+RUN apk add --no-cache ca-certificates openssl
+RUN wget https://github.com/buger/goreplay/releases/download/v0.16.1/gor_0.16.1_x64.tar.gz -O gor.tar.gz
+RUN tar xzf gor.tar.gz

-RUN apt-get update && apt-get install ruby vim-common -y
-
-# Install Java for middleware testing
-RUN echo "deb http://ppa.launchpad.net/webupd8team/java/ubuntu trusty main" | tee /etc/apt/sources.list.d/webupd8team-java.list
-RUN echo "deb-src http://ppa.launchpad.net/webupd8team/java/ubuntu trusty main" | tee -a /etc/apt/sources.list.d/webupd8team-java.list
-RUN apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys EEA14886
-RUN apt-get update -y
-RUN echo oracle-java7-installer shared/accepted-oracle-license-v1-1 select true | /usr/bin/debconf-set-selections
-RUN apt-get install oracle-java8-installer -y
-
-RUN apt-get install flex bison -y
-RUN wget http://www.tcpdump.org/release/libpcap-1.7.4.tar.gz && tar xzf libpcap-1.7.4.tar.gz && cd libpcap-1.7.4 && ./configure && make install
-RUN go get github.com/google/gopacket
-RUN go get -u github.com/golang/lint/golint
-
-WORKDIR /go/src/github.com/buger/gor/
-ADD . /go/src/github.com/buger/gor/
-
-RUN wget http://archive.apache.org/dist/commons/io/binaries/commons-io-2.4-bin.tar.gz && tar xzf commons-io-2.4-bin.tar.gz && cd commons-io-2.4 && mv commons-io-2.4.jar /tmp/
-RUN wget http://archive.apache.org/dist/commons/codec/binaries/commons-codec-1.9-bin.tar.gz && tar xzf commons-codec-1.9-bin.tar.gz
-RUN javac -cp commons-io-2.4/commons-io-2.4.jar -cp commons-codec-1.9/commons-codec-1.9.jar ./examples/middleware/echo.java
-RUN go get
+FROM scratch
+COPY --from=builder /goreplay .
+ENTRYPOINT ["./goreplay"]
@@ -0,0 +1,26 @@
+FROM golang:latest
+
+RUN apt-get update && apt-get install ruby vim-common -y
+
+# Install Java for middleware testing
+RUN echo "deb http://ppa.launchpad.net/webupd8team/java/ubuntu trusty main" | tee /etc/apt/sources.list.d/webupd8team-java.list
+RUN echo "deb-src http://ppa.launchpad.net/webupd8team/java/ubuntu trusty main" | tee -a /etc/apt/sources.list.d/webupd8team-java.list
+RUN apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys EEA14886
+RUN apt-get update -y
+RUN echo oracle-java7-installer shared/accepted-oracle-license-v1-1 select true | /usr/bin/debconf-set-selections
+RUN apt-get install oracle-java8-installer -y
+
+RUN apt-get install flex bison -y
+RUN wget http://www.tcpdump.org/release/libpcap-1.8.1.tar.gz && tar xzf libpcap-1.8.1.tar.gz && cd libpcap-1.8.1 && ./configure && make install
+RUN go get github.com/google/gopacket
+RUN go get -u golang.org/x/lint/golint
+
+WORKDIR /go/src/github.com/buger/goreplay/
+
+RUN wget http://archive.apache.org/dist/commons/io/binaries/commons-io-2.4-bin.tar.gz && tar xzf commons-io-2.4-bin.tar.gz && cd commons-io-2.4 && mv commons-io-2.4.jar /tmp/
+RUN wget http://archive.apache.org/dist/commons/codec/binaries/commons-codec-1.9-bin.tar.gz && tar xzf commons-codec-1.9-bin.tar.gz
+
+ADD . /go/src/github.com/buger/goreplay/
+
+RUN javac -cp commons-io-2.4/commons-io-2.4.jar -cp commons-codec-1.9/commons-codec-1.9.jar ./examples/middleware/echo.java
+RUN go get
@@ -1,5 +1,5 @@
-SOURCE = emitter.go gor.go gor_stat.go input_dummy.go input_file.go input_raw.go input_tcp.go limiter.go output_dummy.go output_null.go output_file.go input_http.go output_http.go output_tcp.go plugins.go settings.go test_input.go elasticsearch.go http_modifier.go http_modifier_settings.go http_client.go middleware.go protocol.go output_file_settings.go output_kafka.go
-SOURCE_PATH = /go/src/github.com/buger/gor/
+SOURCE = $(shell ls -1 *.go | grep -v _test.go)
+SOURCE_PATH = /go/src/github.com/buger/goreplay/
 PORT = 8000
 FADDR = :8000
 RUN = docker run -v `pwd`:$(SOURCE_PATH) -p 0.0.0.0:$(PORT):$(PORT) -i -t gor
@@ -13,19 +13,22 @@ FADDR = ":8000"
 release: release-x64 release-mac

 release-bin:
-	docker run -v `pwd`:$(SOURCE_PATH) -t --env GOOS=linux --env GOARCH=amd64  -i gor go build -tags netgo $(LDFLAGS)
+	docker run -v `pwd`:$(SOURCE_PATH) -t --env GOOS=linux --env GOARCH=amd64  -i gor go build -o gor -tags netgo $(LDFLAGS)

 release-x64:
-	docker run -v `pwd`:$(SOURCE_PATH) -t --env GOOS=linux --env GOARCH=amd64  -i gor go build -tags netgo $(LDFLAGS) && tar -czf gor_$(VERSION)_x64.tar.gz gor && rm gor
+	docker run -v `pwd`:$(SOURCE_PATH) -t --env GOOS=linux --env GOARCH=amd64  -i gor go build -o gor -tags netgo $(LDFLAGS) && tar -czf gor_$(VERSION)_x64.tar.gz gor && rm gor

 release-x86:
-	docker run -v `pwd`:$(SOURCE_PATH) -t --env GOOS=linux --env GOARCH=386 -i gor go build -tags netgo $(LDFLAGS) && tar -czf gor_$(VERSION)_x86.tar.gz gor && rm gor
+	docker run -v `pwd`:$(SOURCE_PATH) -t --env GOOS=linux --env GOARCH=386 -i gor go build -o gor -tags netgo $(LDFLAGS) && tar -czf gor_$(VERSION)_x86.tar.gz gor && rm gor

 release-mac:
-	go build $(MAC_LDFLAGS) && tar -czf gor_$(VERSION)_mac.tar.gz gor && rm gor
+	go build $(MAC_LDFLAGS) -o gor && tar -czf gor_$(VERSION)_mac.tar.gz gor && rm gor
+
+install:
+	go install $(MAC_LDFLAGS)

 build:
-	docker build -t gor .
+	docker build -t gor -f Dockerfile.dev .


 profile:
@@ -68,7 +71,7 @@ run:
 	$(RUN) go run $(LDFLAGS) $(SOURCE) --input-dummy=0 --output-http="http://localhost:9000" --input-raw-track-response --input-raw 127.0.0.1:9000 --verbose --debug --middleware "./examples/middleware/echo.sh" --output-file requests.gor

 run-2:
-	sudo -E go run $(SOURCE) --input-dummy="" --output-tcp localhost:27001 --verbose --debug
+	$(RUN) go run $(LDFLAGS) $(SOURCE) --input-raw :8000 --input-raw-bpf-filter "dst port 8000" --output-stdout --output-http "http://localhost:8000" --input-dummy=0

 run-3:
 	sudo -E go run $(SOURCE) --input-tcp :27001 --output-stdout
@@ -80,7 +83,7 @@ file-server:
 	go run $(SOURCE) file-server $(FADDR)

 readpcap:
-	go run $(SOURCE) --input-raw $(FILE) --input-raw-engine pcap_file --output-null
+	go run $(SOURCE) --input-raw $(FILE) --input-raw-track-response --input-raw-engine pcap_file --output-stdout

 record:
 	$(RUN) go run $(SOURCE) --input-dummy=0 --output-file=requests.gor --verbose --debug
@@ -1,14 +1,16 @@
-[![GitHub release](https://img.shields.io/github/release/buger/gor.svg?maxAge=3600)](https://github.com/buger/gor/releases) [![codebeat](https://codebeat.co/badges/6427d589-a78e-416c-a546-d299b4089893)](https://codebeat.co/projects/github-com-buger-gor) [![Go Report Card](https://goreportcard.com/badge/github.com/buger/gor)](https://goreportcard.com/report/github.com/buger/gor) [![Join the chat at https://gitter.im/buger/gor](https://badges.gitter.im/buger/gor.svg)](https://gitter.im/buger/gor?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)
+[![GitHub release](https://img.shields.io/github/release/buger/gor.svg?maxAge=3600)](https://github.com/buger/goreplay/releases) [![codebeat](https://codebeat.co/badges/6427d589-a78e-416c-a546-d299b4089893)](https://codebeat.co/projects/github-com-buger-gor) [![Go Report Card](https://goreportcard.com/badge/github.com/buger/gor)](https://goreportcard.com/report/github.com/buger/gor) [![Join the chat at https://gitter.im/buger/gor](https://badges.gitter.im/buger/gor.svg)](https://gitter.im/buger/gor?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge) [![Reviewed by Hound](https://img.shields.io/badge/Reviewed_by-Hound-8E64B0.svg)](https://houndci.com)

 ![Go Replay](http://i.imgur.com/ZG2ki5n.png)

 ## https://goreplay.org/

+## We just released a new version, see announcement https://leonsbox.com/goreplay-v0-16-and-4th-anniversary-5408b1fd72e0
+
 ## About

 GoReplay is the simplest and safest way to test your app using real traffic before you put it into production.

-As your application grows, the effort required to test it also grows exponentially. GoReplay offers you the simple idea of reusing your existing traffic for testing, which makes it incredibly powerful. Our state of art technique allows to analyze and record your application traffic without affecting it. This eliminates the risks that come with putting a third party component in the critical path. 
+As your application grows, the effort required to test it also grows exponentially. GoReplay offers you the simple idea of reusing your existing traffic for testing, which makes it incredibly powerful. Our state of art technique allows you to analyze and record your application traffic without affecting it. This eliminates the risks that come with putting a third party component in the critical path. 

 GoReplay increases your confidence in code deployments, configuration changes and infrastructure changes. Did we mention that no coding is required?

@@ -18,17 +20,17 @@ Here is basic workflow: The listener server catches http traffic and sends it to

 ![Diagram](http://i.imgur.com/9mqj2SK.png)

-Check [latest documentation](http://github.com/buger/gor/wiki).
+Check [latest documentation](http://github.com/buger/goreplay/wiki).

 ## Installation
-Download latest binary from https://github.com/buger/gor/releases or [compile by yourself](https://github.com/buger/gor/wiki/Compilation).
+Download latest binary from https://github.com/buger/goreplay/releases or [compile by yourself](https://github.com/buger/goreplay/wiki/Compilation).

 ## Getting started

 The most basic setup will be `sudo ./gor --input-raw :8000 --output-stdout` which acts like tcpdump.
 If you already have test environment you can start replaying: `sudo ./gor --input-raw :8000 --output-http http://staging.env`.

-See the our [documentation](https://github.com/buger/gor/wiki/) and [Getting started](https://github.com/buger/gor/wiki/Getting-Started) page for more info. 
+See the our [documentation](https://github.com/buger/goreplay/wiki/) and [Getting started](https://github.com/buger/goreplay/wiki/Getting-Started) page for more info. 
 ## Newsletter
 Subscribe to our [newsletter](https://www.getdrip.com/forms/89690474/submissions/new) to stay informed about the latest features and changes to Gor project.

@@ -39,9 +41,9 @@ We have created a [GoReplay PRO](https://goreplay.org/pro.html) extension which


 ## Problems?
-If you have a problem, please review the [FAQ](https://github.com/buger/gor/wiki/FAQ) and [Troubleshooting](https://github.com/buger/gor/wiki/Troubleshooting) wiki pages. Searching the [issues](https://github.com/buger/gor/issues) for your problem is also a good idea.
+If you have a problem, please review the [FAQ](https://github.com/buger/goreplay/wiki/FAQ) and [Troubleshooting](https://github.com/buger/goreplay/wiki/Troubleshooting) wiki pages. Searching the [issues](https://github.com/buger/goreplay/issues) for your problem is also a good idea.

-All bug-reports and suggestions should go though Github Issues or our [Google Group](https://groups.google.com/forum/#!forum/gor-users) (you can just send email to gor-users@googlegroups.com).
+All bug-reports and suggestions should go through Github Issues or our [Google Group](https://groups.google.com/forum/#!forum/gor-users) (you can just send email to gor-users@googlegroups.com).
 If you have a private question feel free to send email to support@gortool.com.


@@ -66,7 +68,7 @@ If you have a private question feel free to send email to support@gortool.com.
 * [Granify](http://granify.com) - AI backed SaaS solution that enables online retailers to maximise their sales
 * And many more!

-If you are using Gor we are happy add you to the list and share your story, just write to: hello@goreplay.org
+If you are using Gor, we are happy to add you to the list and share your story, just write to: hello@goreplay.org

 ## Author

@@ -29,12 +29,15 @@ func Replace(a []byte, from, to int, new []byte) []byte {
 		copy(a[from:from+len(new)], new)

 		return a
-	} else if lenDiff < 0 {
+	}
+
+	if lenDiff < 0 {
 		copy(a[from:], new)
 		copy(a[from+len(new):], a[to:])
 		return a[:len(a)+lenDiff]
-	} else { // same size
-		copy(a[from:], new)
-		return a
 	}
+
+	// same size
+	copy(a[from:], new)
+	return a
 }
@@ -13,7 +13,7 @@ If you have multiple replay machines you can split traffic among them using `--s
 gor --input-raw :80 --split-output --output-tcp replay1.local:28020 --output-tcp replay2.local:28020
 ```

-[GoReplay PRO](https://goreplay.com/pro.html) support accurate recording and replaying of tcp sessions, and when `--recognize-tcp-sessions` option is passed, instead of round-robin it will use a smarter algorithm which ensures that same sessions will be sent to the same replay instance.
+[GoReplay PRO](https://goreplay.org/pro.html) support accurate recording and replaying of tcp sessions, and when `--recognize-tcp-sessions` option is passed, instead of round-robin it will use a smarter algorithm which ensures that same sessions will be sent to the same replay instance.


 In case if you are planning a large load testing, you may consider use separate master instance which will control Gor slaves which actually replay traffic. For example:
@@ -26,7 +26,7 @@ Simple bash echo middleware (returns same request) will look like this:
 ```bash
 while read line; do
  echo $line
-end
+done
 ```

 Middleware can be enabled using `--middleware` option, by specifying path to executable file:
@@ -71,4 +71,4 @@ Imagine that you have auth system that randomly generate access tokens, which us

 ***

-You may also read about [[Request filtering]], [[Rate limiting]] and [[Request rewriting]].
+You may also read about [[Request filtering]], [[Rate limiting]] and [[Request rewriting]].
@@ -23,16 +23,16 @@ Set request header, if header already exists it will be overwritten. May be usef

 ```
 gor --input-raw :80 --output-http "http://staging.server" \
-    --http-header "User-Agent: Replayed by Gor" \
-    --http-header "Enable-Feature-X: true"
+    --http-set-header "User-Agent: Replayed by Gor" \
+    --http-set-header "Enable-Feature-X: true"
 ```

 #### Host header
-Host header gets special treatment. By default Host get set to the value specified in --output-http. If you manually set --http-header "Host: anonther.com", Gor will not override Host value.
+Host header gets special treatment. By default Host get set to the value specified in --output-http. If you manually set --http-set-header "Host: anonther.com", Gor will not override Host value.

 If you app accepts traffic from multiple domains, and you want to keep original headers, there is specific `--http-original-host` with tells Gor do not touch Host header at all.


 ***

-You may also read about [[Request filtering]], [[Rate limiting]] and [[Middleware]]
+You may also read about [[Request filtering]], [[Rate limiting]] and [[Middleware]]
@@ -2,17 +2,21 @@ package main

 import (
 	"encoding/json"
-	"github.com/buger/gor/proto"
-	"github.com/mattbaird/elastigo/lib"
 	"log"
-	"regexp"
+	"net/url"
+	"strings"
+	//"regexp"
 	"time"
+
+	"github.com/buger/goreplay/proto"
+
+	"github.com/mattbaird/elastigo/lib"
 )

 type ESUriErorr struct{}

 func (e *ESUriErorr) Error() string {
-	return "Wrong ElasticSearch URL format. Expected to be: host:port/index_name"
+	return "Wrong ElasticSearch URL format. Expected to be: scheme://host/index_name"
 }

 type ESPlugin struct {
@@ -52,17 +56,27 @@ type ESRequestResponse struct {

 // Parse ElasticSearch URI
 //
-// Proper format is: host:port/index_name
-func parseURI(URI string) (err error, host string, port string, index string) {
-	rURI := regexp.MustCompile("(.+):([0-9]+)/(.+)")
-	match := rURI.FindAllStringSubmatch(URI, -1)
+// Proper format is: scheme://[userinfo@]host/index_name
+// userinfo is: user[:password]
+// net/url.Parse() does not fail if scheme is not provided but actualy does not
+// handle URI properly.
+// So we must 'validate' URI format to match requirements to use net/url.Parse()
+func parseURI(URI string) (err error, index string) {

-	if len(match) == 0 {
+	parsedUrl, parseErr := url.Parse(URI)
+
+	if parseErr != nil {
+		err = new(ESUriErorr)
+	}
+
+	//	check URL validity by extracting host and undex values.
+	host := parsedUrl.Host
+	urlPathParts := strings.Split(parsedUrl.Path, "/")
+	index = urlPathParts[len(urlPathParts)-1]
+
+	// force index specification in uri : ie no implicit index
+	if host == "" || index == "" {
 		err = new(ESUriErorr)
-	} else {
-		host = match[0][1]
-		port = match[0][2]
-		index = match[0][3]
 	}

 	return
@@ -71,14 +85,15 @@ func parseURI(URI string) (err error, host string, port string, index string) {
 func (p *ESPlugin) Init(URI string) {
 	var err error

-	err, p.Host, p.ApiPort, p.Index = parseURI(URI)
+	err, p.Index = parseURI(URI)

 	if err != nil {
 		log.Fatal("Can't initialize ElasticSearch plugin.", err)
 	}
+
 	p.eConn = elastigo.NewConn()
-	p.eConn.SetPort(p.ApiPort)
-	p.eConn.SetHosts([]string{p.Host})
+
+	p.eConn.SetFromUrl(URI)

 	p.indexor = p.eConn.NewBulkIndexerErrors(50, 60)
 	p.done = make(chan bool)
@@ -0,0 +1,140 @@
+package main
+
+import (
+	"testing"
+)
+
+const expectedIndex = "gor"
+
+func assertExpectedGorIndex(index string, t *testing.T) {
+	if expectedIndex != index {
+		t.Fatalf("Expected index %s but got %s", expectedIndex, index)
+	}
+}
+
+func assertExpectedIndex(expectedIndex string, index string, t *testing.T) {
+	if expectedIndex != index {
+		t.Fatalf("Expected index %s but got %s", expectedIndex, index)
+	}
+}
+
+func assertExpectedError(returnedError error, t *testing.T) {
+	expectedError := new(ESUriErorr)
+
+	if expectedError != returnedError {
+		t.Errorf("Expected err %s but got %s", expectedError, returnedError)
+	}
+}
+
+func assertNoError(returnedError error, t *testing.T) {
+	if nil != returnedError {
+		t.Errorf("Expected no err but got %s", returnedError)
+	}
+}
+
+// Argument host:port/index_name
+// i.e : localhost:9200/gor
+// Fail because scheme is mandatory
+func TestElasticConnectionBuildFailWithoutScheme(t *testing.T) {
+	uri := "localhost:9200/" + expectedIndex
+
+	err, _ := parseURI(uri)
+	assertExpectedError(err, t)
+}
+
+// Argument scheme://host:port
+// i.e : http://localhost:9200
+// Fail : explicit index is required
+func TestElasticConnectionBuildFailWithoutIndex(t *testing.T) {
+	uri := "http://localhost:9200"
+
+	err, index := parseURI(uri)
+
+	assertExpectedIndex("", index, t)
+
+	assertExpectedError(err, t)
+}
+
+// Argument scheme://host/index_name
+// i.e : http://localhost/gor
+func TestElasticConnectionBuildFailWithoutPort(t *testing.T) {
+	uri := "http://localhost/" + expectedIndex
+
+	err, index := parseURI(uri)
+
+	assertNoError(err, t)
+
+	assertExpectedGorIndex(index, t)
+}
+
+// Argument scheme://host:port/index_name
+// i.e : http://localhost:9200/gor
+func TestElasticLocalConnectionBuild(t *testing.T) {
+	uri := "http://localhost:9200/" + expectedIndex
+
+	err, index := parseURI(uri)
+
+	assertNoError(err, t)
+
+	assertExpectedGorIndex(index, t)
+}
+
+// Argument scheme://host:port/index_name
+// i.e : http://localhost.local:9200/gor or https://localhost.local:9200/gor
+func TestElasticSimpleLocalWithSchemeConnectionBuild(t *testing.T) {
+	uri := "http://localhost.local:9200/" + expectedIndex
+
+	err, index := parseURI(uri)
+
+	assertNoError(err, t)
+
+	assertExpectedGorIndex(index, t)
+}
+
+// Argument scheme://host:port/index_name
+// i.e : http://localhost.local:9200/gor or https://localhost.local:9200/gor
+func TestElasticSimpleLocalWithHTTPSConnectionBuild(t *testing.T) {
+	uri := "https://localhost.local:9200/" + expectedIndex
+
+	err, index := parseURI(uri)
+
+	assertNoError(err, t)
+
+	assertExpectedGorIndex(index, t)
+}
+
+// Argument scheme://host:port/index_name
+// i.e : localhost.local:9200/pathtoElastic/gor
+func TestElasticLongPathConnectionBuild(t *testing.T) {
+	uri := "http://localhost.local:9200/pathtoElastic/" + expectedIndex
+
+	err, index := parseURI(uri)
+
+	assertNoError(err, t)
+
+	assertExpectedGorIndex(index, t)
+}
+
+// Argument scheme://host:userinfo@port/index_name
+// i.e : http://user:password@localhost.local:9200/gor
+func TestElasticBasicAuthConnectionBuild(t *testing.T) {
+	uri := "http://user:password@localhost.local:9200/" + expectedIndex
+
+	err, index := parseURI(uri)
+
+	assertNoError(err, t)
+
+	assertExpectedGorIndex(index, t)
+}
+
+// Argument scheme://host:port/path/index_name
+// i.e : http://localhost.local:9200/path/gor or https://localhost.local:9200/path/gor
+func TestElasticComplexPathConnectionBuild(t *testing.T) {
+	uri := "http://localhost.local:9200/path/" + expectedIndex
+
+	err, index := parseURI(uri)
+
+	assertNoError(err, t)
+
+	assertExpectedGorIndex(index, t)
+}
@@ -3,6 +3,7 @@ package main
 import (
 	"bytes"
 	"io"
+	"log"
 	"time"
 )

@@ -22,10 +23,31 @@ func Start(stop chan int) {
 			}
 		}

-		go CopyMulty(middleware, Plugins.Outputs...)
+		go func() {
+			if err := CopyMulty(middleware, Plugins.Outputs...); err != nil {
+				log.Println("Error during copy: ", err)
+				close(stop)
+			}
+		}()
 	} else {
 		for _, in := range Plugins.Inputs {
-			go CopyMulty(in, Plugins.Outputs...)
+			go func() {
+				if err := CopyMulty(in, Plugins.Outputs...); err != nil {
+					log.Println("Error during copy: ", err)
+					close(stop)
+				}
+			}()
+		}
+
+		for _, out := range Plugins.Outputs {
+			if r, ok := out.(io.Reader); ok {
+				go func() {
+					if err := CopyMulty(r, Plugins.Outputs...); err != nil {
+						log.Println("Error during copy: ", err)
+						close(stop)
+					}
+				}()
+			}
 		}
 	}

@@ -41,7 +63,7 @@ func Start(stop chan int) {

 // CopyMulty copies from 1 reader to multiple writers
 func CopyMulty(src io.Reader, writers ...io.Writer) (err error) {
-	buf := make([]byte, 5*1024*1024)
+	buf := make([]byte, Settings.copyBufferSize)
 	wIndex := 0
 	modifier := NewHTTPModifier(&Settings.modifierConfig)
 	filteredRequests := make(map[string]time.Time)
@@ -52,14 +74,30 @@ func CopyMulty(src io.Reader, writers ...io.Writer) (err error) {
 	for {
 		nr, er := src.Read(buf)

+		if er == io.EOF {
+			return nil
+		}
+		if er != nil {
+			return err
+		}
+
+		_maxN := nr
+		if nr > 500 {
+			_maxN = 500
+		}
 		if nr > 0 && len(buf) > nr {
 			payload := buf[:nr]
 			meta := payloadMeta(payload)
+			if len(meta) < 3 {
+				if Settings.debug {
+					Debug("[EMITTER] Found malformed record", string(payload[0:_maxN]), nr, "from:", src)
+				}
+				continue
+			}
 			requestID := string(meta[1])

-			_maxN := nr
-			if nr > 500 {
-				_maxN = 500
+			if nr >= 5*1024*1024 {
+				log.Println("INFO: Large packet... We received ", len(payload), " bytes from ", src)
 			}

 			if Settings.debug {
@@ -88,15 +126,24 @@ func CopyMulty(src io.Reader, writers ...io.Writer) (err error) {
 					}
 				} else {
 					if _, ok := filteredRequests[requestID]; ok {
-						delete(filteredRequests, requestID);
+						delete(filteredRequests, requestID)
 						continue
 					}
 				}
 			}

+			if Settings.prettifyHTTP {
+				payload = prettifyHTTP(payload)
+				if len(payload) == 0 {
+					continue
+				}
+			}
+
 			if Settings.splitOutput {
 				// Simple round robin
-				writers[wIndex].Write(payload)
+				if _, err := writers[wIndex].Write(payload); err != nil {
+					return err
+				}

 				wIndex++

@@ -105,26 +152,22 @@ func CopyMulty(src io.Reader, writers ...io.Writer) (err error) {
 				}
 			} else {
 				for _, dst := range writers {
-					dst.Write(payload)
+					if _, err := dst.Write(payload); err != nil {
+						return err
+					}
 				}
 			}
-
-		}
-		if er == io.EOF {
-			break
-		}
-		if er != nil {
-			err = er
-			break
+		} else if nr > 0 {
+			log.Println("WARN: Packet", nr, "bytes is too large to process. Consider increasing --copy-buffer-size")
 		}

 		// Run GC on each 1000 request
-		if i % 1000 == 0 {
+		if i%1000 == 0 {
 			// Clean up filtered requests for which we didn't get a response to filter
 			now := time.Now()
-			if now.Sub(filteredRequestsLastCleanTime) > 60 * time.Second {
+			if now.Sub(filteredRequestsLastCleanTime) > 60*time.Second {
 				for k, v := range filteredRequests {
-					if now.Sub(v) > 60 * time.Second {
+					if now.Sub(v) > 60*time.Second {
 						delete(filteredRequests, k)
 					}
 				}
@@ -75,6 +75,8 @@ func TestEmitterFiltered(t *testing.T) {
 	wg.Wait()

 	close(quit)
+
+	Settings.modifierConfig = HTTPModifierConfig{}
 }

 func TestEmitterRoundRobin(t *testing.T) {
@@ -0,0 +1,57 @@
+#!/usr/bin/env node
+const readline = require("readline");
+const StringDecoder = require("string_decoder").StringDecoder
+
+const rl = readline.createInterface({
+  input: process.stdin
+});
+
+var ignoreIds = new Set();
+var ignoreAddresses = "/api";
+const decoder = new StringDecoder("utf8");
+
+function convertHexString(hex) {
+  var bytes = [];
+  for (var i = 0; i < hex.length - 1; i += 2) {
+    bytes.push(parseInt(hex.substr(i, 2), 16));
+  }
+  return decoder.write(Buffer.from(bytes));
+}
+
+function log(output) {
+	console.error("===================");
+	console.error(output);
+}
+
+function shouldOutputLine(request) {
+  const components = request.split("\n");
+  const header = components[0].split(" ");
+  const type = parseInt(header[0]);
+  const tag = header[1];
+
+  if (type === 3) {
+    return true;
+  }
+  if (type === 1) {
+    // Check if it's oauth
+    const endpoint = components[1].split(" ")[1];
+    if (!endpoint.startsWith(ignoreAddresses)) {
+      ignoreIds.add(tag);
+      return false;
+    }
+  } else if (type === 2) {
+    if (ignoreIds.has(tag)) {
+      ignoreIds.delete(tag);
+      return false;
+    }
+  }
+  return true;
+}
+
+rl.on("line", (input) => {
+  const str = convertHexString(input);
+  console.log(input);
+  if (shouldOutputLine(str)) {
+    log(str);
+	}
+});
@@ -0,0 +1,76 @@
+#! /usr/bin/env python3
+# -*- coding: utf-8 -*-
+
+import sys
+import fileinput
+import binascii
+
+# Used to find end of the Headers section
+EMPTY_LINE = b'\r\n\r\n'
+
+
+def log(msg):
+    """
+    Logging to STDERR as STDOUT and STDIN used for data transfer
+    @type msg: str or byte string
+    @param msg: Message to log to STDERR
+    """
+    try:
+        msg = str(msg) + '\n'
+    except:
+        pass
+    sys.stderr.write(msg)
+    sys.stderr.flush()
+
+
+def find_end_of_headers(byte_data):
+    """
+    Finds where the header portion ends and the content portion begins.
+    @type byte_data: str or byte string
+    @param byte_data: Hex decoded req or resp string
+    """
+    return byte_data.index(EMPTY_LINE) + 4
+
+
+def process_stdin():
+    """
+    Process STDIN and output to STDOUT
+    """
+    for raw_line in fileinput.input():
+
+        line = raw_line.rstrip()
+
+        # Decode base64 encoded line
+        decoded = bytes.fromhex(line)
+
+        # Split into metadata and payload, the payload is headers + body
+        (raw_metadata, payload) = decoded.split(b'\n', 1)
+
+        # Split into headers and payload
+        headers_pos = find_end_of_headers(payload)
+        raw_headers = payload[:headers_pos]
+        raw_content = payload[headers_pos:]
+
+        log('===================================')
+        request_type_id = int(raw_metadata.split(b' ')[0])
+        log('Request type: {}'.format({
+          1: 'Request',
+          2: 'Original Response',
+          3: 'Replayed Response'
+        }[request_type_id]))
+        log('===================================')
+
+        log('Original data:')
+        log(line)
+
+        log('Decoded request:')
+        log(decoded)
+
+        encoded = binascii.hexlify(raw_metadata + b'\n' + raw_headers + raw_content).decode('ascii')
+        log('Encoded data:')
+        log(encoded)
+
+        sys.stdout.write(encoded + '\n')
+
+if __name__ == '__main__':
+    process_stdin()
@@ -1,12 +1,12 @@
 #!/usr/bin/env ruby
 # encoding: utf-8
-while data = STDIN.gets # continiously read line from STDIN
+while data = STDIN.gets # continuously read line from STDIN
  next unless data
  data = data.chomp # remove end of line symbol
  
  decoded = [data].pack("H*") # decode base64 encoded request
  
-  # dedoded value is raw HTTP payload, example:
+  # decoded value is raw HTTP payload, example:
  #   
  #   POST /post HTTP/1.1
  #   Content-Length: 7
@@ -24,8 +24,9 @@ import (
 	"bytes"
 	"encoding/hex"
 	"fmt"
-	"github.com/buger/gor/proto"
 	"os"
+
+	"github.com/buger/goreplay/proto"
 )

 // requestID -> originalToken
@@ -9,6 +9,7 @@ import (
 	"log"
 	"net/http"
 	"net/http/httputil"
+	_ "net/http/pprof"
 	"os"
 	"os/signal"
 	"runtime"
@@ -32,7 +33,10 @@ func loggingMiddleware(next http.Handler) http.Handler {
 	})
 }

+var closeCh chan int
+
 func main() {
+	closeCh = make(chan int)
 	// // Don't exit on panic
 	// defer func() {
 	// 	if r := recover(); r != nil {
@@ -52,7 +56,7 @@ func main() {
 		}
 		dir, _ := os.Getwd()

-		log.Println("Started example file server for current dirrectory on address ", args[1])
+		log.Println("Started example file server for current directory on address ", args[1])

 		log.Fatal(http.ListenAndServe(args[1], loggingMiddleware(http.FileServer(http.Dir(dir)))))
 	} else {
@@ -74,6 +78,12 @@ func main() {
 		profileCPU(*cpuprofile)
 	}

+	if Settings.pprof != "" {
+		go func() {
+			log.Println(http.ListenAndServe(Settings.pprof, nil))
+		}()
+	}
+
 	c := make(chan os.Signal, 1)
 	signal.Notify(c, os.Interrupt, syscall.SIGTERM)
 	go func() {
@@ -84,17 +94,14 @@ func main() {

 	if Settings.exitAfter > 0 {
 		log.Println("Running gor for a duration of", Settings.exitAfter)
-		closeCh := make(chan int)

 		time.AfterFunc(Settings.exitAfter, func() {
 			log.Println("Stopping gor after", Settings.exitAfter)
 			close(closeCh)
 		})
-
-		Start(closeCh)
-	} else {
-		Start(nil)
 	}
+
+	Start(closeCh)
 }

 func finalize() {
@@ -7,21 +7,19 @@ import (
 	"time"
 )

-const (
-	rate = 5
-)
-
 type GorStat struct {
 	statName string
+	rateMs   int
 	latest   int
 	mean     int
 	max      int
 	count    int
 }

-func NewGorStat(statName string) (s *GorStat) {
+func NewGorStat(statName string, rateMs int) (s *GorStat) {
 	s = new(GorStat)
 	s.statName = statName
+	s.rateMs = rateMs
 	s.latest = 0
 	s.mean = 0
 	s.max = 0
@@ -40,7 +38,7 @@ func (s *GorStat) Write(latest int) {
 			s.max = latest
 		}
 		if latest != 0 {
-			s.mean = (s.mean + latest) / 2
+			s.mean = ((s.mean * s.count) + latest) / (s.count + 1)
 		}
 		s.latest = latest
 		s.count = s.count + 1
@@ -55,13 +53,13 @@ func (s *GorStat) Reset() {
 }

 func (s *GorStat) String() string {
-	return s.statName + ":" + strconv.Itoa(s.latest) + "," + strconv.Itoa(s.mean) + "," + strconv.Itoa(s.max) + "," + strconv.Itoa(s.count) + "," + strconv.Itoa(s.count/rate) + "," + strconv.Itoa(runtime.NumGoroutine())
+	return s.statName + ":" + strconv.Itoa(s.latest) + "," + strconv.Itoa(s.mean) + "," + strconv.Itoa(s.max) + "," + strconv.Itoa(s.count) + "," + strconv.Itoa(s.count/(s.rateMs/1000.0)) + "," + strconv.Itoa(runtime.NumGoroutine())
 }

 func (s *GorStat) reportStats() {
 	for {
 		log.Println(s)
 		s.Reset()
-		time.Sleep(rate * time.Second)
+		time.Sleep(time.Duration(s.rateMs) * time.Millisecond)
 	}
 }
@@ -1,12 +1,14 @@
 package main

 import (
+	"bufio"
 	"bytes"
 	"crypto/tls"
 	"encoding/base64"
 	"io"
 	"log"
 	"net"
+	"net/http"
 	"net/url"
 	"runtime/debug"
 	"strconv"
@@ -15,7 +17,7 @@ import (
 	"syscall"
 	"time"

-	"github.com/buger/gor/proto"
+	"github.com/buger/goreplay/proto"
 )

 var httpMu sync.Mutex
@@ -47,6 +49,8 @@ type HTTPClient struct {
 	host           string
 	auth           string
 	conn           net.Conn
+	proxy          *url.URL
+	proxyAuth      string
 	respBuf        []byte
 	config         *HTTPClientConfig
 	redirectsCount int
@@ -61,7 +65,7 @@ func NewHTTPClient(baseURL string, config *HTTPClientConfig) *HTTPClient {

 	if config.Timeout == 0 {
 		config.Timeout = time.Second
-    }
+	}

 	config.ConnectionTimeout = config.Timeout

@@ -80,19 +84,74 @@ func NewHTTPClient(baseURL string, config *HTTPClientConfig) *HTTPClient {
 		client.auth = "Basic " + base64.StdEncoding.EncodeToString([]byte(u.User.String()))
 	}

+	client.proxy, _ = http.ProxyFromEnvironment(&http.Request{URL: u})
+
+	if client.isProxy() && client.proxy.User != nil {
+		client.proxyAuth = "Basic " + base64.StdEncoding.EncodeToString([]byte(client.proxy.User.String()))
+	}
+
 	return client
 }

 func (c *HTTPClient) Connect() (err error) {
 	c.Disconnect()

+	var toDial string
 	if !strings.Contains(c.host, ":") {
-		c.conn, err = net.DialTimeout("tcp", c.host + ":" + defaultPorts[c.scheme], c.config.ConnectionTimeout)
+		toDial = c.host + ":" + defaultPorts[c.scheme]
 	} else {
-		c.conn, err = net.DialTimeout("tcp", c.host, c.config.ConnectionTimeout)
+		toDial = c.host
+	}
+
+	if c.isProxy() {
+		if c.proxy.Scheme != "http" {
+			panic("Unsupported HTTP Proxy method")
+		}
+		Debug("[HTTPClient] Connecting to proxy", c.proxy.String(), "<>", toDial)
+		c.conn, err = net.DialTimeout("tcp", c.proxy.Host, c.config.ConnectionTimeout)
+		if err != nil {
+			return
+		}
+		if c.scheme == "https" {
+			c.conn.Write([]byte("CONNECT " + toDial + " HTTP/1.1\r\n"))
+			if c.proxyAuth != "" {
+				c.conn.Write([]byte("Proxy-Authorization: " + c.proxyAuth + "\r\n"))
+			}
+			c.conn.Write([]byte("\r\n"))
+			br := bufio.NewReader(c.conn)
+			l, _, err := br.ReadLine()
+			if err != nil {
+				return err
+			}
+			if len(l) < 12 {
+				panic("HTTP proxy did not respond correctly")
+			}
+			status := l[9:12]
+			if !bytes.Equal(status, []byte("200")) {
+				panic("HTTP proxy did not respond correctly")
+			}
+			for {
+				// Read until we find the empty line
+				l, _, err := br.ReadLine()
+				if err != nil {
+					return err
+				}
+				if len(l) == 0 {
+					break
+				}
+			}
+		}
+		Debug("[HTTPClient] Proxy successfully connected")
+	} else {
+		c.conn, err = net.DialTimeout("tcp", toDial, c.config.ConnectionTimeout)
+		if err != nil {
+			return
+		}
 	}

 	if c.scheme == "https" {
+		// Wrap our socket in TLS
+		Debug("[HTTPClient] Wrapping socket in TLS", c.host)
 		tlsConn := tls.Client(c.conn, &tls.Config{InsecureSkipVerify: true, ServerName: c.host})

 		if err = tlsConn.Handshake(); err != nil {
@@ -100,6 +159,7 @@ func (c *HTTPClient) Connect() (err error) {
 		}

 		c.conn = tlsConn
+		Debug("[HTTPClient] Successfully wrapped in TLS")
 	}

 	return
@@ -113,25 +173,24 @@ func (c *HTTPClient) Disconnect() {
 	}
 }

-func (c *HTTPClient) isAlive() bool {
-	one := make([]byte, 1)
-
+func (c *HTTPClient) isAlive(readBytes *int) bool {
 	// Ready 1 byte from socket without timeout to check if it not closed
 	c.conn.SetReadDeadline(time.Now().Add(time.Millisecond))
-	_, err := c.conn.Read(one)
+	n, err := c.conn.Read(c.respBuf[:1])

-	if err == nil {
-		return true
-	} else if err == io.EOF {
-		if c.config.Debug {
-			Debug("[HTTPClient] connection closed, reconnecting")
-		}
-		return false
-	} else if err == syscall.EPIPE {
-		Debug("Detected broken pipe.", err)
+	if err == io.EOF {
+		Debug("[HTTPClient] connection closed, reconnecting")
 		return false
 	}

+	if err == syscall.EPIPE {
+		Debug("Detected broken pipe.", err)
+		return false
+	}
+	if n != 0 {
+		*readBytes += n
+		Debug("[HTTPClient] isAlive readBytes ", *readBytes)
+	}
 	return true
 }

@@ -141,14 +200,16 @@ func (c *HTTPClient) Send(data []byte) (response []byte, err error) {
 		if r := recover(); r != nil {
 			Debug("[HTTPClient]", r, string(data))

-			if _, ok := r.(error); !ok {
+			if _, ok := r.(error); ok {
 				log.Println("[HTTPClient] Failed to send request: ", string(data))
-				log.Println("PANIC: pkg:", r, debug.Stack())
+				log.Println("[HTTPClient] Response: ", string(response))
+				log.Println("PANIC: pkg:", r, string(debug.Stack()))
 			}
 		}
 	}()

-	if c.conn == nil || !c.isAlive() {
+	var readBytes int
+	if c.conn == nil || !c.isAlive(&readBytes) {
 		Debug("[HTTPClient] Connecting:", c.baseURL)
 		if err = c.Connect(); err != nil {
 			log.Println("[HTTPClient] Connection error:", err)
@@ -165,6 +226,16 @@ func (c *HTTPClient) Send(data []byte) (response []byte, err error) {
 		data = proto.SetHost(data, []byte(c.baseURL), []byte(c.host))
 	}

+	if c.isProxy() && c.scheme == "http" {
+		path := proto.Path(data)
+		if len(path) > 0 && path[0] == '/' {
+			data = proto.SetPath(data, c.proxyPath(path))
+			if c.proxyAuth != "" {
+				data = proto.SetHeader(data, []byte("Proxy-Authorization"), []byte(c.proxyAuth))
+			}
+		}
+	}
+
 	if c.auth != "" {
 		data = proto.SetHeader(data, []byte("Authorization"), []byte(c.auth))
 	}
@@ -173,13 +244,19 @@ func (c *HTTPClient) Send(data []byte) (response []byte, err error) {
 		Debug("[HTTPClient] Sending:", string(data))
 	}

+	return c.send(data, readBytes, timeout)
+}
+
+func (c *HTTPClient) send(data []byte, readBytes int, timeout time.Time) (response []byte, err error) {
+	var payload []byte
+	var n int
 	if _, err = c.conn.Write(data); err != nil {
 		Debug("[HTTPClient] Write error:", err, c.baseURL)
 		response = errorPayload(HTTP_TIMEOUT)
+		c.Disconnect()
 		return
 	}

-	var readBytes, n int
 	var currentChunk []byte
 	timeout = time.Now().Add(c.config.Timeout)
 	chunked := false
@@ -195,27 +272,30 @@ func (c *HTTPClient) Send(data []byte) (response []byte, err error) {
 			readBytes += n
 			chunks++

-			if err != nil {
-				if err == io.EOF {
-					err = nil
-				}
-				break
-			}
-
 			// First chunk
 			if chunked || contentLength != -1 {
 				currentContentLength += n
 			} else {
 				// If headers are finished
-				if bytes.Contains(c.respBuf[:readBytes], proto.EmptyLine) {
-					if bytes.Equal(proto.Header(c.respBuf, []byte("Transfer-Encoding")), []byte("chunked")) {
+				var firstEmptyLine = bytes.Index(c.respBuf[:readBytes], proto.EmptyLine)
+				if firstEmptyLine != -1 {
+					if bytes.Equal(proto.Header(c.respBuf[:readBytes], []byte("Transfer-Encoding")), []byte("chunked")) {
 						chunked = true
 					} else {
-						status, _ := strconv.Atoi(string(proto.Status(c.respBuf)))
-						if (status >= 100 && status < 200) || status == 204 || status == 304 {
+						status, _ := strconv.Atoi(string(proto.Status(c.respBuf[:readBytes])))
+						// We want to soak up all 100 Continues received to get the real result code
+						if status >= 100 && status < 200 {
+							timeout = time.Now().Add(c.config.Timeout)
+							var deleteLen = firstEmptyLine + len(proto.EmptyLine)
+							copy(c.respBuf, c.respBuf[deleteLen:readBytes])
+							readBytes -= deleteLen
+							chunks--
+							continue
+						} else if status == 204 || status == 304 {
 							contentLength = 0
+							break
 						} else {
-							l := proto.Header(c.respBuf, []byte("Content-Length"))
+							l := proto.Header(c.respBuf[:readBytes], []byte("Content-Length"))
 							if len(l) > 0 {
 								contentLength, _ = strconv.Atoi(string(l))
 							}
@@ -240,6 +320,13 @@ func (c *HTTPClient) Send(data []byte) (response []byte, err error) {
 					break
 				}
 			}
+
+			if err != nil {
+				if err == io.EOF {
+					err = nil
+				}
+				break
+			}
 		} else {
 			if currentChunk == nil {
 				currentChunk = make([]byte, readChunkSize)
@@ -247,13 +334,6 @@ func (c *HTTPClient) Send(data []byte) (response []byte, err error) {

 			n, err = c.conn.Read(currentChunk)

-			if err == io.EOF {
-				break
-			} else if err != nil {
-				Debug("[HTTPClient] Read the whole body error:", err, c.baseURL)
-				break
-			}
-
 			readBytes += int(n)
 			chunks++
 			currentContentLength += n
@@ -276,6 +356,14 @@ func (c *HTTPClient) Send(data []byte) (response []byte, err error) {
 				c.Disconnect()
 				break
 			}
+
+			if err == io.EOF {
+				break
+			} else if err != nil {
+				Debug("[HTTPClient] Read the whole body error:", err, c.baseURL)
+				break
+			}
+
 		}

 		if readBytes >= maxResponseSize {
@@ -288,16 +376,32 @@ func (c *HTTPClient) Send(data []byte) (response []byte, err error) {
 		timeout = time.Now().Add(c.config.Timeout / 5)
 	}

-	if err != nil {
-		Debug("[HTTPClient] Response read error", err, c.conn, readBytes)
+	if err != nil && readBytes == 0 {
+		maxRead := 100
+		if readBytes < maxRead {
+			maxRead = readBytes
+		}
+		Debug("[HTTPClient] Response read timeout error", err, c.conn, readBytes, string(c.respBuf[:maxRead]))
 		response = errorPayload(HTTP_TIMEOUT)
+		c.Disconnect()
+		return
+	}
+
+	if readBytes < 4 || string(c.respBuf[:4]) != "HTTP" {
+		maxRead := 100
+		if readBytes < maxRead {
+			maxRead = readBytes
+		}
+		Debug("[HTTPClient] Response read unknown error", err, c.conn, readBytes, string(c.respBuf[:maxRead]))
+		response = errorPayload(HTTP_UNKNOWN_ERROR)
+		c.Disconnect()
 		return
 	}

 	if readBytes > len(c.respBuf) {
 		readBytes = len(c.respBuf)
 	}
-	payload := make([]byte, readBytes)
+	payload = make([]byte, readBytes)
 	copy(payload, c.respBuf[:readBytes])

 	if c.config.Debug {
@@ -312,7 +416,7 @@ func (c *HTTPClient) Send(data []byte) (response []byte, err error) {
 			c.redirectsCount++

 			location := proto.Header(payload, []byte("Location"))
-			redirectPayload := []byte("GET " + string(location) + " HTTP/1.1\r\n\r\n")
+			redirectPayload := proto.SetPath(data, location)

 			if c.config.Debug {
 				Debug("[HTTPClient] Redirecting to: " + string(location))
@@ -323,8 +427,8 @@ func (c *HTTPClient) Send(data []byte) (response []byte, err error) {
 	}

 	if bytes.Equal(proto.Status(payload), []byte("400")) {
-		c.Disconnect()
 		Debug("[HTTPClient] Closed connection on 400 response")
+		c.Disconnect()
 	}

 	c.redirectsCount = 0
@@ -346,6 +450,14 @@ func (c *HTTPClient) Post(path string, body []byte) (response []byte, err error)
 	return c.Send([]byte(payload))
 }

+func (c *HTTPClient) proxyPath(path []byte) []byte {
+	return append([]byte(c.scheme+"://"+c.host), path...)
+}
+
+func (c *HTTPClient) isProxy() bool {
+	return c.proxy != nil
+}
+
 const (
 	// https://support.cloudflare.com/hc/en-us/articles/200171936-Error-520-Web-server-is-returning-an-unknown-error
 	HTTP_UNKNOWN_ERROR = "520"
@@ -3,7 +3,6 @@ package main
 import (
 	"bytes"
 	"crypto/rand"
-	"github.com/buger/gor/proto"
 	"io/ioutil"
 	_ "log"
 	"net"
@@ -15,6 +14,8 @@ import (
 	"sync"
 	"testing"
 	"time"
+
+	"github.com/buger/goreplay/proto"
 )

 func TestHTTPClientURLPort(t *testing.T) {
@@ -357,6 +358,35 @@ func TestHTTPClientRedirectLimit(t *testing.T) {
 	wg.Wait()
 }

+func TestHTTPClientKeepHeadersRedirect(t *testing.T) {
+	wg := new(sync.WaitGroup)
+
+	GETPayload := []byte("GET / HTTP/1.1\r\n\r\n")
+
+	GETPayload = proto.AddHeader(GETPayload, []byte("keep-header"), []byte("true"))
+
+	server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+
+		if r.URL.Path == "/" {
+			http.Redirect(w, r, "/new", 301)
+		}
+
+		if r.Header.Get("keep-header") != "true" {
+			t.Errorf("Header keep-header was incorrect, got: %s, want: %s.", r.Header.Get("keep-header"), "true")
+		}
+
+		wg.Done()
+	}))
+	defer server.Close()
+
+	client := NewHTTPClient(server.URL, &HTTPClientConfig{FollowRedirects: 1, Debug: false})
+
+	wg.Add(2)
+	client.Send(GETPayload)
+
+	wg.Wait()
+}
+
 func TestHTTPClientBasicAuth(t *testing.T) {
 	wg := new(sync.WaitGroup)
 	wg.Add(2)
@@ -2,9 +2,11 @@ package main

 import (
 	"bytes"
+	"encoding/base64"
 	"hash/fnv"
+	"strings"

-	"github.com/buger/gor/proto"
+	"github.com/buger/goreplay/proto"
 )

 type HTTPModifier struct {
@@ -16,8 +18,10 @@ func NewHTTPModifier(config *HTTPModifierConfig) *HTTPModifier {
 	if len(config.urlRegexp) == 0 &&
 		len(config.urlNegativeRegexp) == 0 &&
 		len(config.urlRewrite) == 0 &&
+		len(config.headerRewrite) == 0 &&
 		len(config.headerFilters) == 0 &&
 		len(config.headerNegativeFilters) == 0 &&
+		len(config.headerBasicAuthFilters) == 0 &&
 		len(config.headerHashFilters) == 0 &&
 		len(config.paramHashFilters) == 0 &&
 		len(config.params) == 0 &&
@@ -94,7 +98,11 @@ func (m *HTTPModifier) Rewrite(payload []byte) (response []byte) {
 		for _, f := range m.config.headerFilters {
 			value := proto.Header(payload, f.name)

-			if len(value) > 0 && !f.regexp.Match(value) {
+			if len(value) == 0 {
+				return
+			}
+
+			if !f.regexp.Match(value) {
 				return
 			}
 		}
@@ -110,6 +118,23 @@ func (m *HTTPModifier) Rewrite(payload []byte) (response []byte) {
 		}
 	}

+	if len(m.config.headerBasicAuthFilters) > 0 {
+		for _, f := range m.config.headerBasicAuthFilters {
+			value := proto.Header(payload, []byte("Authorization"))
+
+			if len(value) > 0 {
+				valueString := string(value)
+				trimmedBasicAuthEncoded := strings.TrimPrefix(valueString, "Basic ")
+				if strings.Compare(valueString, trimmedBasicAuthEncoded) != 0 {
+					decodedAuth, _ := base64.StdEncoding.DecodeString(trimmedBasicAuthEncoded)
+					if !f.regexp.Match(decodedAuth) {
+						return
+					}
+				}
+			}
+		}
+	}
+
 	if len(m.config.headerHashFilters) > 0 {
 		for _, f := range m.config.headerHashFilters {
 			value := proto.Header(payload, f.name)
@@ -153,5 +178,19 @@ func (m *HTTPModifier) Rewrite(payload []byte) (response []byte) {
 		}
 	}

+	if len(m.config.headerRewrite) > 0 {
+		for _, f := range m.config.headerRewrite {
+			value := proto.Header(payload, f.header)
+			if len(value) == 0 {
+				break
+			}
+
+			if f.src.Match(value) {
+				newValue := f.src.ReplaceAll(value, f.target)
+				payload = proto.SetHeader(payload, f.header, newValue)
+			}
+		}
+	}
+
 	return payload
 }
@@ -10,13 +10,15 @@ import (

 // HTTPModifierConfig holds configuration options for built-in traffic modifier
 type HTTPModifierConfig struct {
-	urlNegativeRegexp     HTTPUrlRegexp
-	urlRegexp             HTTPUrlRegexp
-	urlRewrite            UrlRewriteMap
-	headerFilters         HTTPHeaderFilters
-	headerNegativeFilters HTTPHeaderFilters
-	headerHashFilters     HTTPHashFilters
-	paramHashFilters      HTTPHashFilters
+	urlNegativeRegexp      HTTPUrlRegexp
+	urlRegexp              HTTPUrlRegexp
+	urlRewrite             UrlRewriteMap
+	headerRewrite          HeaderRewriteMap
+	headerFilters          HTTPHeaderFilters
+	headerNegativeFilters  HTTPHeaderFilters
+	headerBasicAuthFilters HTTPHeaderBasicAuthFilters
+	headerHashFilters      HTTPHashFilters
+	paramHashFilters       HTTPHashFilters

 	params  HTTPParams
 	headers HTTPHeaders
@@ -54,6 +56,31 @@ func (h *HTTPHeaderFilters) Set(value string) error {
 	return nil
 }

+//
+// Handling of --http-basic-auth-filter option
+//
+type basicAuthFilter struct {
+	regexp *regexp.Regexp
+}
+
+// HTTPHeaderFilters holds list of headers and their regexps
+type HTTPHeaderBasicAuthFilters []basicAuthFilter
+
+func (h *HTTPHeaderBasicAuthFilters) String() string {
+	return fmt.Sprint(*h)
+}
+
+func (h *HTTPHeaderBasicAuthFilters) Set(value string) error {
+	r, err := regexp.Compile(value)
+	if err != nil {
+		return err
+	}
+
+	*h = append(*h, basicAuthFilter{regexp: r})
+
+	return nil
+}
+
 //
 // Handling of --http-allow-header-hash and --http-allow-param-hash options
 //
@@ -196,6 +223,42 @@ func (r *UrlRewriteMap) Set(value string) error {
 	return nil
 }

+//
+// Handling of --http-rewrite-header option
+//
+type headerRewrite struct {
+	header []byte
+	src    *regexp.Regexp
+	target []byte
+}
+
+type HeaderRewriteMap []headerRewrite
+
+func (r *HeaderRewriteMap) String() string {
+	return fmt.Sprint(*r)
+}
+
+func (r *HeaderRewriteMap) Set(value string) error {
+	headerArr := strings.SplitN(value, ":", 2)
+	if len(headerArr) < 2 {
+		return errors.New("need both header, regexp and rewrite target, colon-delimited (ex. Header: regexp,target)")
+	}
+
+	header := headerArr[0]
+	valArr := strings.SplitN(strings.TrimSpace(headerArr[1]), ",", 2)
+
+	if len(valArr) < 2 {
+		return errors.New("need both header, regexp and rewrite target, colon-delimited (ex. Header: regexp,target)")
+	}
+
+	regexp, err := regexp.Compile(valArr[0])
+	if err != nil {
+		return err
+	}
+	*r = append(*r, headerRewrite{header: []byte(header), src: regexp, target: []byte(valArr[1])})
+	return nil
+}
+
 //
 // Handling of --http-allow-url option
 //
@@ -2,8 +2,9 @@ package main

 import (
 	"bytes"
-	"github.com/buger/gor/proto"
 	"testing"
+
+	"github.com/buger/goreplay/proto"
 )

 func TestHTTPModifierWithoutConfig(t *testing.T) {
@@ -78,6 +79,45 @@ func TestHTTPModifierHeaderNegativeFilters(t *testing.T) {
 	}
 }

+func TestHTTPHeaderBasicAuthFilters(t *testing.T) {
+	filters := HTTPHeaderBasicAuthFilters{}
+	filters.Set("^customer[0-9].*")
+
+	modifier := NewHTTPModifier(&HTTPModifierConfig{
+		headerBasicAuthFilters: filters,
+	})
+
+	//Encoded UserId:Password = customer3:welcome
+	payload := []byte("POST /post HTTP/1.1\r\nContent-Length: 7\r\nAuthorization: Basic Y3VzdG9tZXIzOndlbGNvbWU=\r\n\r\na=1&b=2")
+	if len(modifier.Rewrite(payload)) == 0 {
+		t.Error("Request should pass filters")
+	}
+
+	//customer6:rest@123^TEST
+	payload = []byte("POST /post HTTP/1.1\r\nContent-Length: 88\r\nAuthorization: Basic Y3VzdG9tZXI2OnJlc3RAMTIzXlRFU1Q==\r\n\r\na=1&b=2")
+	if len(modifier.Rewrite(payload)) == 0 {
+		t.Error("Request should pass filters")
+	}
+
+	filters = HTTPHeaderBasicAuthFilters{}
+	// Setting filter that not match our header
+	filters.Set("^(homer simpson|mickey mouse).*")
+
+	modifier = NewHTTPModifier(&HTTPModifierConfig{
+		headerBasicAuthFilters: filters,
+	})
+
+	if len(modifier.Rewrite(payload)) != 0 {
+		t.Error("Request should not pass filters")
+	}
+
+	//mickey mouse:happy123
+	payload = []byte("POST /post HTTP/1.1\r\nContent-Length: 88\r\nAuthorization: Basic bWlja2V5IG1vdXNlOmhhcHB5MTIz\r\n\r\na=1&b=2")
+	if len(modifier.Rewrite(payload)) == 0 {
+		t.Error("Request should pass filters")
+	}
+}
+
 func TestHTTPModifierURLRewrite(t *testing.T) {
 	var url, newURL []byte

@@ -107,6 +147,27 @@ func TestHTTPModifierURLRewrite(t *testing.T) {
 	}
 }

+func TestHTTPModifierHeaderRewrite(t *testing.T) {
+	var header, newHeader []byte
+
+	rewrites := HeaderRewriteMap{}
+	payload := []byte("GET / HTTP/1.1\r\nContent-Length: 7\r\nHost: www.w3.org\r\n\r\na=1&b=2")
+
+	err := rewrites.Set("Host: (.*).w3.org,$1.beta.w3.org")
+	if err != nil {
+		t.Error("Should not error", err)
+	}
+
+	modifier := NewHTTPModifier(&HTTPModifierConfig{
+		headerRewrite: rewrites,
+	})
+
+	header = []byte("www.beta.w3.org")
+	if newHeader = proto.Header(modifier.Rewrite(payload), []byte("Host")); !bytes.Equal(newHeader, header) {
+		t.Error("Request header should have been rewritten, wasn't", string(newHeader), string(header))
+	}
+}
+
 func TestHTTPModifierHeaderHashFilters(t *testing.T) {
 	filters := HTTPHashFilters{}
 	filters.Set("Header2:1/2")
@@ -0,0 +1,75 @@
+package main
+
+import (
+	"bytes"
+	"compress/gzip"
+	"io/ioutil"
+	"net/http/httputil"
+	"strconv"
+
+	"github.com/buger/goreplay/proto"
+)
+
+func prettifyHTTP(p []byte) []byte {
+	headSize := bytes.IndexByte(p, '\n') + 1
+	head := p[:headSize]
+	body := p[headSize:]
+
+	headersPos := proto.MIMEHeadersEndPos(body)
+
+	if headersPos < 5 || headersPos > len(body) {
+		return p
+	}
+
+	headers := body[:headersPos]
+	content := body[headersPos:]
+
+	var tEnc, cEnc []byte
+	proto.ParseHeaders([][]byte{headers}, func(header, value []byte) bool {
+		if proto.HeadersEqual(header, []byte("Transfer-Encoding")) {
+			tEnc = value
+		}
+
+		if proto.HeadersEqual(header, []byte("Content-Encoding")) {
+			cEnc = value
+		}
+
+		return true
+	})
+
+	if len(tEnc) == 0 && len(cEnc) == 0 {
+		return p
+	}
+
+	if bytes.Equal(tEnc, []byte("chunked")) {
+		buf := bytes.NewBuffer(content)
+		r := httputil.NewChunkedReader(buf)
+		content, _ = ioutil.ReadAll(r)
+
+		headers = proto.DeleteHeader(headers, []byte("Transfer-Encoding"))
+
+		newLen := strconv.Itoa(len(content))
+		headers = proto.SetHeader(headers, []byte("Content-Length"), []byte(newLen))
+	}
+
+	if bytes.Equal(cEnc, []byte("gzip")) {
+		buf := bytes.NewBuffer(content)
+		g, err := gzip.NewReader(buf)
+
+		if err != nil {
+			Debug("[Prettifier] GZIP encoding error:", err)
+			return []byte{}
+		}
+
+		content, _ = ioutil.ReadAll(g)
+
+		headers = proto.DeleteHeader(headers, []byte("Content-Encoding"))
+
+		newLen := strconv.Itoa(len(content))
+		headers = proto.SetHeader(headers, []byte("Content-Length"), []byte(newLen))
+	}
+
+	newPayload := append(append(head, headers...), content...)
+
+	return newPayload
+}
@@ -0,0 +1,36 @@
+package main
+
+import (
+	"bytes"
+	"compress/gzip"
+	"strconv"
+	"testing"
+)
+
+func TestHTTPPrettifierGzip(t *testing.T) {
+	b := bytes.NewBufferString("")
+	w := gzip.NewWriter(b)
+	w.Write([]byte("test"))
+	w.Close()
+
+	size := strconv.Itoa(len(b.Bytes()))
+
+	payload := []byte("2 1 1\nHTTP/1.1 200 OK\r\nContent-Length: " + size + "\r\nContent-Encoding: gzip\r\n\r\n")
+	payload = append(payload, b.Bytes()...)
+
+	newPayload := prettifyHTTP(payload)
+
+	if string(newPayload) != "2 1 1\nHTTP/1.1 200 OK\r\nContent-Length: 4\r\n\r\ntest" {
+		t.Error("Payload not match:", string(newPayload))
+	}
+}
+
+func TestHTTPPrettifierChunked(t *testing.T) {
+	payload := []byte("POST / HTTP/1.1\r\nHost: www.w3.org\r\nTransfer-Encoding: chunked\r\n\r\n4\r\nWiki\r\n5\r\npedia\r\ne\r\n in\r\n\r\nchunks.\r\n0\r\n\r\n")
+
+	newPayload := prettifyHTTP(payload)
+
+	if string(newPayload) != "POST / HTTP/1.1\r\nHost: www.w3.org\r\nContent-Length: 23\r\n\r\nWikipedia in\r\n\r\nchunks." {
+		t.Error("Payload not match:", string(newPayload))
+	}
+}
@@ -222,6 +222,13 @@ func (i *FileInput) emit() {
 	}

 	log.Printf("FileInput: end of file '%s'\n", i.path)
+
+	// For now having fixed timeout is temporary solution
+	// Further should be modified, so outputs can report if their queue empty or not
+	time.Sleep(time.Second)
+	if closeCh != nil {
+		close(closeCh)
+	}
 }

 func (i *FileInput) Close() error {
@@ -1,7 +1,6 @@
 package main

 import (
-	"github.com/buger/gor/proto"
 	"io"
 	"log"
 	"net/http"
@@ -9,6 +8,8 @@ import (
 	"strings"
 	"sync"
 	"testing"
+
+	"github.com/buger/goreplay/proto"
 )

 func TestHTTPInput(t *testing.T) {
@@ -2,9 +2,11 @@ package main

 import (
 	"encoding/json"
+	"log"
+	"strings"
+
 	"github.com/Shopify/sarama"
 	"github.com/Shopify/sarama/mocks"
-	"log"
 )

 // KafkaInput is used for recieving Kafka messages and
@@ -22,11 +24,12 @@ func NewKafkaInput(address string, config *KafkaConfig) *KafkaInput {

 	var con sarama.Consumer

-	if config.consumer.(*mocks.Consumer) != nil {
+	if mock, ok := config.consumer.(*mocks.Consumer); ok && mock != nil {
 		con = config.consumer
 	} else {
 		var err error
-		con, err = sarama.NewConsumer([]string{config.host}, c)
+		//con, err = sarama.NewConsumer([]string{config.host}, c)
+		con, err = sarama.NewConsumer(strings.Split(config.host, ","), c)

 		if err != nil {
 			log.Fatalln("Failed to start Sarama(Kafka) consumer:", err)
@@ -82,20 +85,21 @@ func (i *KafkaInput) Read(data []byte) (int, error) {
 	if !i.config.useJSON {
 		copy(data, message.Value)
 		return len(message.Value), nil
-	} else {
-		var kafkaMessage KafkaMessage
-		json.Unmarshal(message.Value, &kafkaMessage)
-
-		buf, err := kafkaMessage.Dump()
-		if err != nil {
-			log.Println("Failed to decode access log entry:", err)
-			return 0, err
-		}
-
-		copy(data, buf)
-
-		return len(buf), nil
 	}
+
+	var kafkaMessage KafkaMessage
+	json.Unmarshal(message.Value, &kafkaMessage)
+
+	buf, err := kafkaMessage.Dump()
+	if err != nil {
+		log.Println("Failed to decode access log entry:", err)
+		return 0, err
+	}
+
+	copy(data, buf)
+
+	return len(buf), nil
+
 }

 func (i *KafkaInput) String() string {
@@ -1,9 +1,10 @@
 package main

 import (
+	"testing"
+
 	"github.com/Shopify/sarama"
 	"github.com/Shopify/sarama/mocks"
-	"testing"
 )

 func TestInputKafkaRAW(t *testing.T) {
@@ -12,7 +13,7 @@ func TestInputKafkaRAW(t *testing.T) {

 	consumer.ExpectConsumePartition("test", 0, mocks.AnyOffset).YieldMessage(&sarama.ConsumerMessage{Value: []byte("1 2 3\nGET / HTTP1.1\r\nHeader: 1\r\n\r\n")})
 	consumer.SetTopicMetadata(
-		map[string][]int32{"test": []int32{0}},
+		map[string][]int32{"test": {0}},
 	)

 	input := NewKafkaInput("", &KafkaConfig{
@@ -39,7 +40,7 @@ func TestInputKafkaJSON(t *testing.T) {

 	consumer.ExpectConsumePartition("test", 0, mocks.AnyOffset).YieldMessage(&sarama.ConsumerMessage{Value: []byte(`{"Req_URL":"/","Req_Type":"1","Req_ID":"2","Req_Ts":"3","Req_Method":"GET","Req_Headers":{"Header":"1"}}`)})
 	consumer.SetTopicMetadata(
-		map[string][]int32{"test": []int32{0}},
+		map[string][]int32{"test": {0}},
 	)

 	input := NewKafkaInput("", &KafkaConfig{
@@ -1,11 +1,12 @@
 package main

 import (
-	"github.com/buger/gor/proto"
-	raw "github.com/buger/gor/raw_socket_listener"
 	"log"
 	"net"
 	"time"
+
+	"github.com/buger/goreplay/proto"
+	raw "github.com/buger/goreplay/raw_socket_listener"
 )

 // RAWInput used for intercepting traffic for given address
@@ -18,6 +19,9 @@ type RAWInput struct {
 	realIPHeader  []byte
 	trackResponse bool
 	listener      *raw.Listener
+	bpfFilter     string
+	timestampType string
+	bufferSize    int
 }

 // Available engines for intercepting traffic
@@ -28,15 +32,18 @@ const (
 )

 // NewRAWInput constructor for RAWInput. Accepts address with port as argument.
-func NewRAWInput(address string, engine int, trackResponse bool, expire time.Duration, realIPHeader string) (i *RAWInput) {
+func NewRAWInput(address string, engine int, trackResponse bool, expire time.Duration, realIPHeader string, bpfFilter string, timestampType string, bufferSize int) (i *RAWInput) {
 	i = new(RAWInput)
 	i.data = make(chan *raw.TCPMessage)
 	i.address = address
 	i.expire = expire
 	i.engine = engine
+	i.bpfFilter = bpfFilter
 	i.realIPHeader = []byte(realIPHeader)
 	i.quit = make(chan bool)
 	i.trackResponse = trackResponse
+	i.timestampType = timestampType
+	i.bufferSize = bufferSize

 	i.listen(address)
 	i.listener.IsReady()
@@ -56,7 +63,7 @@ func (i *RAWInput) Read(data []byte) (int, error) {
 			buf = proto.SetHeader(buf, i.realIPHeader, []byte(msg.IP().String()))
 		}
 	} else {
-		header = payloadHeader(ResponsePayload, msg.UUID(), msg.AssocMessage.Start.UnixNano(), msg.End.UnixNano()-msg.AssocMessage.Start.UnixNano())
+		header = payloadHeader(ResponsePayload, msg.UUID(), msg.Start.UnixNano(), msg.End.UnixNano()-msg.AssocMessage.End.UnixNano())
 	}

 	copy(data[0:len(header)], header)
@@ -70,17 +77,11 @@ func (i *RAWInput) listen(address string) {

 	host, port, err := net.SplitHostPort(address)

-	if i.engine == EnginePcapFile {
-		host = address
-		port = "1"
-		err = nil
-	}
-
 	if err != nil {
 		log.Fatal("input-raw: error while parsing address", err)
 	}

-	i.listener = raw.NewListener(host, port, i.engine, i.trackResponse, i.expire)
+	i.listener = raw.NewListener(host, port, i.engine, i.trackResponse, i.expire, i.bpfFilter, i.timestampType, i.bufferSize, Settings.inputRAWOverrideSnapLen, Settings.inputRAWImmediateMode)

 	ch := i.listener.Receiver()

@@ -2,7 +2,6 @@ package main

 import (
 	"bytes"
-	"github.com/buger/gor/proto"
 	"io"
 	"io/ioutil"
 	"log"
@@ -19,6 +18,8 @@ import (
 	"sync/atomic"
 	"testing"
 	"time"
+
+	"github.com/buger/goreplay/proto"
 )

 const testRawExpire = time.Millisecond * 200
@@ -43,7 +44,7 @@ func TestRAWInputIPv4(t *testing.T) {

 	var respCounter, reqCounter int64

-	input := NewRAWInput(originAddr, EnginePcap, true, testRawExpire, "X-Real-IP")
+	input := NewRAWInput(originAddr, EnginePcap, true, testRawExpire, "X-Real-IP", "", "", 0)
 	defer input.Close()

 	output := NewTestOutput(func(data []byte) {
@@ -105,7 +106,7 @@ func TestRAWInputNoKeepAlive(t *testing.T) {

 	originAddr := listener.Addr().String()

-	input := NewRAWInput(originAddr, EnginePcap, true, testRawExpire, "")
+	input := NewRAWInput(originAddr, EnginePcap, true, testRawExpire, "", "", "", 0)
 	defer input.Close()

 	output := NewTestOutput(func(data []byte) {
@@ -151,7 +152,7 @@ func TestRAWInputIPv6(t *testing.T) {

 	var respCounter, reqCounter int64

-	input := NewRAWInput(originAddr, EnginePcap, true, testRawExpire, "")
+	input := NewRAWInput(originAddr, EnginePcap, true, testRawExpire, "", "", "", 0)
 	defer input.Close()

 	output := NewTestOutput(func(data []byte) {
@@ -202,7 +203,7 @@ func TestInputRAW100Expect(t *testing.T) {

 	originAddr := strings.Replace(origin.Listener.Addr().String(), "[::]", "127.0.0.1", -1)

-	input := NewRAWInput(originAddr, EnginePcap, true, time.Second, "")
+	input := NewRAWInput(originAddr, EnginePcap, true, time.Second, "", "", "", 0)
 	defer input.Close()

 	// We will use it to get content of raw HTTP request
@@ -265,7 +266,7 @@ func TestInputRAWChunkedEncoding(t *testing.T) {
 	}))

 	originAddr := strings.Replace(origin.Listener.Addr().String(), "[::]", "127.0.0.1", -1)
-	input := NewRAWInput(originAddr, EnginePcap, true, time.Second, "")
+	input := NewRAWInput(originAddr, EnginePcap, true, time.Second, "", "", "", 0)
 	defer input.Close()

 	replay := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
@@ -329,7 +330,7 @@ func TestInputRAWLargePayload(t *testing.T) {
 	}))
 	originAddr := strings.Replace(origin.Listener.Addr().String(), "[::]", "127.0.0.1", -1)

-	input := NewRAWInput(originAddr, EnginePcap, true, testRawExpire, "")
+	input := NewRAWInput(originAddr, EnginePcap, true, testRawExpire, "", "", "", 0)
 	defer input.Close()

 	replay := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, req *http.Request) {
@@ -374,7 +375,7 @@ func BenchmarkRAWInput(b *testing.B) {

 	var respCounter, reqCounter int64

-	input := NewRAWInput(originAddr, EnginePcap, true, testRawExpire, "")
+	input := NewRAWInput(originAddr, EnginePcap, true, testRawExpire, "", "", "", 0)
 	defer input.Close()

 	output := NewTestOutput(func(data []byte) {
@@ -3,6 +3,7 @@ package main
 import (
 	"bufio"
 	"bytes"
+	"crypto/tls"
 	"fmt"
 	"io"
 	"log"
@@ -13,15 +14,23 @@ import (
 // TCPInput used for internal communication
 type TCPInput struct {
 	data     chan []byte
-	address  string
 	listener net.Listener
+	address  string
+	config   *TCPInputConfig
+}
+
+type TCPInputConfig struct {
+	secure          bool
+	certificatePath string
+	keyPath         string
 }

 // NewTCPInput constructor for TCPInput, accepts address with port
-func NewTCPInput(address string) (i *TCPInput) {
+func NewTCPInput(address string, config *TCPInputConfig) (i *TCPInput) {
 	i = new(TCPInput)
 	i.data = make(chan []byte, 1000)
 	i.address = address
+	i.config = config

 	i.listen(address)

@@ -36,16 +45,30 @@ func (i *TCPInput) Read(data []byte) (int, error) {
 }

 func (i *TCPInput) listen(address string) {
-	listener, err := net.Listen("tcp", address)
-	i.listener = listener
+	if i.config.secure {
+		cer, err := tls.LoadX509KeyPair(i.config.certificatePath, i.config.keyPath)
+		if err != nil {
+			log.Fatal("Error while loading --input-file certificate:", err)
+		}

-	if err != nil {
-		log.Fatal("Can't start:", err)
+		config := &tls.Config{Certificates: []tls.Certificate{cer}}
+		listener, err := tls.Listen("tcp", address, config)
+		if err != nil {
+			log.Fatal("Can't start --input-tcp with secure connection:", err)
+		}
+		i.listener = listener
+	} else {
+		listener, err := net.Listen("tcp", address)
+		if err != nil {
+			log.Fatal("Can't start:", err)
+		}
+
+		i.listener = listener
 	}

 	go func() {
 		for {
-			conn, err := listener.Accept()
+			conn, err := i.listener.Accept()

 			if err != nil {
 				log.Println("Error while Accept()", err)
@@ -1,18 +1,28 @@
 package main

 import (
+	"bytes"
+	"crypto/rand"
+	"crypto/rsa"
+	"crypto/tls"
+	"crypto/x509"
+	"encoding/pem"
 	"io"
+	"io/ioutil"
 	"log"
+	"math/big"
 	"net"
+	"os"
 	"sync"
 	"testing"
+	"time"
 )

 func TestTCPInput(t *testing.T) {
 	wg := new(sync.WaitGroup)
 	quit := make(chan int)

-	input := NewTCPInput("127.0.0.1:0")
+	input := NewTCPInput("127.0.0.1:0", &TCPInputConfig{})
 	output := NewTestOutput(func(data []byte) {
 		wg.Done()
 	})
@@ -46,3 +56,81 @@ func TestTCPInput(t *testing.T) {

 	close(quit)
 }
+
+func genCertificate(template *x509.Certificate) ([]byte, []byte) {
+	priv, _ := rsa.GenerateKey(rand.Reader, 2048)
+
+	serialNumberLimit := new(big.Int).Lsh(big.NewInt(1), 128)
+	serialNumber, _ := rand.Int(rand.Reader, serialNumberLimit)
+	template.SerialNumber = serialNumber
+	template.BasicConstraintsValid = true
+	template.NotBefore = time.Now()
+	template.NotAfter = time.Now().Add(time.Hour)
+
+	derBytes, _ := x509.CreateCertificate(rand.Reader, template, template, &priv.PublicKey, priv)
+
+	var certPem, keyPem bytes.Buffer
+	pem.Encode(&certPem, &pem.Block{Type: "CERTIFICATE", Bytes: derBytes})
+	pem.Encode(&keyPem, &pem.Block{Type: "RSA PRIVATE KEY", Bytes: x509.MarshalPKCS1PrivateKey(priv)})
+
+	return certPem.Bytes(), keyPem.Bytes()
+}
+
+func TestTCPInputSecure(t *testing.T) {
+	serverCertPem, serverPrivPem := genCertificate(&x509.Certificate{
+		DNSNames:    []string{"localhost"},
+		IPAddresses: []net.IP{net.ParseIP("127.0.0.1"), net.ParseIP("::")},
+	})
+
+	serverCertPemFile, _ := ioutil.TempFile("", "server.crt")
+	serverCertPemFile.Write(serverCertPem)
+	serverCertPemFile.Close()
+
+	serverPrivPemFile, _ := ioutil.TempFile("", "server.key")
+	serverPrivPemFile.Write(serverPrivPem)
+	serverPrivPemFile.Close()
+
+	defer func() {
+		os.Remove(serverPrivPemFile.Name())
+		os.Remove(serverCertPemFile.Name())
+	}()
+
+	wg := new(sync.WaitGroup)
+	quit := make(chan int)
+
+	input := NewTCPInput("127.0.0.1:0", &TCPInputConfig{
+		secure:          true,
+		certificatePath: serverCertPemFile.Name(),
+		keyPath:         serverPrivPemFile.Name(),
+	})
+	output := NewTestOutput(func(data []byte) {
+		wg.Done()
+	})
+
+	Plugins.Inputs = []io.Reader{input}
+	Plugins.Outputs = []io.Writer{output}
+
+	go Start(quit)
+
+	conf := &tls.Config{
+		InsecureSkipVerify: true,
+	}
+
+	conn, err := tls.Dial("tcp", input.listener.Addr().String(), conf)
+	if err != nil {
+		t.Fatal(err)
+	}
+	defer conn.Close()
+
+	msg := []byte("1 1 1\nGET / HTTP/1.1\r\n\r\n")
+
+	for i := 0; i < 100; i++ {
+		wg.Add(1)
+		conn.Write(msg)
+		conn.Write([]byte(payloadSeparator))
+	}
+
+	wg.Wait()
+
+	close(quit)
+}
@@ -3,8 +3,10 @@ package main
 import (
 	"bytes"
 	"fmt"
+
+	"github.com/buger/goreplay/proto"
+
 	"github.com/Shopify/sarama"
-	"github.com/buger/gor/proto"
 )

 // KafkaConfig should contains required information to
@@ -82,7 +82,11 @@ func (l *Limiter) Write(data []byte) (n int, err error) {
 }

 func (l *Limiter) Read(data []byte) (n int, err error) {
-	n, err = l.plugin.(io.Reader).Read(data)
+	if r, ok := l.plugin.(io.Reader); ok {
+		n, err = r.Read(data)
+	} else {
+		return 0, nil
+	}

 	if l.isLimited() {
 		return 0, nil
@@ -92,5 +96,5 @@ func (l *Limiter) Read(data []byte) (n int, err error) {
 }

 func (l *Limiter) String() string {
-	return fmt.Sprintf("Limiting %s to: %d (isPercent: %b)", l.plugin, l.limit, l.isPercent)
+	return fmt.Sprintf("Limiting %s to: %d (isPercent: %v)", l.plugin, l.limit, l.isPercent)
 }
@@ -34,9 +34,7 @@ func NewMiddleware(command string) *Middleware {
 	m.Stdout, _ = cmd.StdoutPipe()
 	m.Stdin, _ = cmd.StdinPipe()

-	if Settings.verbose {
-		cmd.Stderr = os.Stderr
-	}
+	cmd.Stderr = os.Stderr

 	go m.read(m.Stdout)

@@ -64,23 +62,40 @@ func (m *Middleware) ReadFrom(plugin io.Reader) {

 func (m *Middleware) copy(to io.Writer, from io.Reader) {
 	buf := make([]byte, 5*1024*1024)
-	dst := make([]byte, len(buf)*2)
+	dst := make([]byte, len(buf)*4)

 	for {
 		nr, _ := from.Read(buf)
-		if nr > 0 && len(buf) > nr {
+		if nr == 0 || nr > len(buf) {
+			continue
+		}

-			hex.Encode(dst, buf[0:nr])
-			dst[nr*2] = '\n'
+		payload := buf[0:nr]

-			m.mu.Lock()
-			to.Write(dst[0 : nr*2+1])
-			m.mu.Unlock()
+		if Settings.prettifyHTTP {
+			payload = prettifyHTTP(payload)
+			nr = len(payload)

-			if Settings.debug {
-				Debug("[MIDDLEWARE-MASTER] Sending:", string(buf[0:nr]), "From:", from)
+			if nr*2 > len(dst) {
+				continue
 			}
 		}
+
+		if Settings.prettifyHTTP {
+			payload = prettifyHTTP(payload)
+			nr = len(payload)
+		}
+
+		hex.Encode(dst, payload)
+		dst[nr*2] = '\n'
+
+		m.mu.Lock()
+		to.Write(dst[0 : nr*2+1])
+		m.mu.Unlock()
+
+		if Settings.debug {
+			Debug("[MIDDLEWARE-MASTER] Sending:", string(buf[0:nr]), "From:", from)
+		}
 	}
 }

@@ -0,0 +1,121 @@
+# GoReplay middleware
+
+GoReplay support protocol for writing middleware in any language, which allows you to implement custom logic like authentification or complex rewriting and filterting. See protocol description here: https://github.com/buger/goreplay/wiki/Middleware, but the basic idea that middleware process receive hex encoded data via STDIN and emits it back via STDOUT. STDERR for loggin inside middleware. Yes, that's simple.
+
+To simplify middleware creation we provide packages for NodeJS and Go (upcoming).
+
+If you want to get access to original and replayed responses, do not forget adding `--output-http-track-respose` and `--input-raw-track-response` options.
+
+## NodeJS
+
+Before starting, you should install the package via npm: `npm install goreplay_middleware`.
+And initialize middleware the following way:
+```javascript
+var gor = require("goreplay_middleware");
+// `init` will initialize STDIN listener
+gor.init();
+```
+
+Basic idea is that you write callbacks which respond to `request`, `response`, `replay`, or `message` events, which contain request meta information and actuall http paylod. Depending on your needs you may compare, override or filter incoming requests and responses.
+
+You can respond to the incoming events using `on` function, by providing callbacks:
+```javascript
+// valid events are `request`, `response` (original response), `replay` (replayed response), and `message` (all events)
+gor.on('request', function(data) {
+    // `data` contains incoming message its meta information.
+    data
+
+    // Raw HTTP payload of `Buffer` type
+    // Example (hidden character for line endings shown on purpose):
+    //   GET / HTTP/1.1\r\n
+    //   User-Agent: Golang\r\n
+    //   \r\n
+    data.http
+
+    // Meta is an array size of 4, containing:
+    //   1. request type - 1, 2 or 3 (which maps to `request`, `respose` and `replay`)
+    //   2. uuid - request unique identifier. Request responses have the same ID as their request.
+    //   3. timestamp of when request was made (for responses it is time of request start too)
+    //   4. latency - time difference between request start and finish. For `request` is zero.
+    data.meta
+
+    // Unique request ID. It should be same for `request`, `response` and `replay` events of the same request.
+    data.ID
+
+    // You should return data at the end of function, even if you not changed request, if you do not want to filter it out.
+    // If you just `return` nothing, request will be filtered
+    return data
+})
+```
+### Mapping requests and responses
+You can provide request ID as additional argument to `on` function, which allow you to map related requests and responses. Below is example of middleware which checks that original and replayed response have same HTTP status code.
+
+```javascript
+// Example of very basic way to compare if replayed traffic have no errors
+gor.on("request", function(req) {
+    gor.on("response", req.ID, function(resp) {
+        gor.on("replay", req.ID, function(repl) {
+            if (gor.httpStatus(resp.http) != gor.httpStatus(repl.http)) {
+                // Note that STDERR is used for logging, and it actually will be send to `Gor` STDOUT.
+                // This trick is used because SDTIN and STDOUT already used for process communication.
+                // You can write logger that writes to files insead.
+                console.error(`${gor.httpPath(req.http)} STATUS NOT MATCH: 'Expected ${gor.httpStatus(resp.http)}' got '${gor.httpStatus(repl.http)}'`)
+            }
+            return repl;
+        })
+        return resp;
+    })
+    return req
+})
+```
+
+This middleware include `searchResponses` helper used to compare values from original and replayed responses. It may be helpful if auth system or xsrf protection returns unique tokens in headers or response, and you need to rewrite your requests based on them. Because tokens are unique, value contained in original and replayed response will differ, so you need to extract value from both responses, and rewrite requests based on those mappings.
+
+`searchResponses` accepts request id, regexp pattern for searching the compared value (should include capture group), and callback which returns both original and replayed matched value.
+
+Example: 
+```javascript
+   // Compare HTTP headers for response and replayed response, and map values
+let tokMap = {};
+
+gor.on("request", function(req) {
+    let tok = gor.httpHeader(req.http, "Auth-Token");
+    if (tok && tokMap[tok]) {
+        req.http = gor.setHttpHeader(req.http, "Auth-Token", tokMap[tok]) 
+    }
+    
+    gor.searchResponses(req.ID, "X-Set-Token: (\w+)$", function(respTok, replTok) {
+        if (respTok && replTok) tokMap[respTok] = replTok;
+    })
+
+    return req;
+})
+```
+
+
+### API documentation
+
+Package expose following functions to process raw HTTP payloads:
+* `init` - initialize middleware object, start reading from STDIN.
+* `httpPath` - URL path of the request: `gor.httpPath(req.http)`
+* `httpMethod` - Http method: 'GET', 'POST', etc. `gor.httpMethod(req.http)`. 
+* `setHttpPath` - update URL path: `req.http = gor.setHttpPath(req.http, newPath)`
+* `httpPathParam` - get param from URL path: `gor.httpPathParam(req.http, queryParam)`
+* `setHttpPathParam` - set URL param: `req.http = gor.setHttpPathParam(req.http, queryParam, value)` 
+* `httpStatus` - response status code
+* `httpHeaders` - get all headers: `gor.httpHeaders(req.http)`
+* `httpHeader` - get HTTP header: `gor.httpHeader(req.http, "Content-Length")`
+* `setHttpHeader` - Set HTTP header, returns modified payload: `req.http = gor.setHttpHeader(req.http, "X-Replayed", "1")`
+* `httpBody` - get HTTP Body: `gor.httpBody(req.http)`
+* `setHttpBody` - Set HTTP Body and ensures that `Content-Length` header have proper value. Returns modified payload: `req.http = gor.setHttpBody(req.http, Buffer.from('hello!'))`.
+* `httpBodyParam` - get POST body param: `gor.httpBodyParam(req.http, param)`
+* `setHttpBodyParam` - set POST body param: `req.http = gor.setHttpBodyParam(req.http, param, value)`
+* `httpCookie` - get HTTP cookie: `gor.httpCookie(req.http, "SESSSION_ID")`
+* `setHttpCookie` - set HTTP cookie, returns modified payload: `req.http = gor.setHttpCookie(req.http, "iam", "cuckoo")`
+
+Also it is totally legit to use standard `Buffer` functions like `indexOf` for processing the HTTP payload. Just do not forget that if you modify modify the body, update the `Content-Length` header with new value. And if you modify headers, line endings should be `\r\n`. Rest is up to your imagination.
+
+
+## Support
+
+Feel free to ask questions here and by sending email to [support@goreplay.org](mailto:support@goreplay.org). Commercial support available and welcomed 🙈.
@@ -0,0 +1,762 @@
+// ======= GoReplay Middleware helper =============
+// Created by Leonid Bugaev in 2017
+//
+// For questions use GitHub or support@goreplay.org
+//
+// GoReplay: https://github.com/buger/goreplay
+// Middleware package: https://github.com/buger/goreplay/middleware
+
+var middleware;
+
+function init() {
+    var proxy = {
+        ch: {},
+        on: function(chan, id, cb) {
+            if (!cb && id) {
+                cb = id;
+            } else if (cb && id) {
+                chan = chan + "#" + id;
+            }
+
+            if (!proxy.ch[chan]) {
+                proxy.ch[chan] = [];
+            }
+
+            proxy.ch[chan].push({
+                created: new Date(),
+                cb: cb
+            });
+
+            return proxy;
+        },
+
+        emit: function(msg, raw) {
+            var chanPrefix;
+
+            switch(msg.type) {
+                case "1": chanPrefix = "request"; break;
+                case "2": chanPrefix = "response"; break;
+                case "3": chanPrefix = "replay"; break;
+            }
+
+            let resp = msg;
+
+            ["message", chanPrefix, chanPrefix + "#" + msg.ID].forEach(function(chanID, idx){
+                if (proxy.ch[chanID]) {
+                    proxy.ch[chanID].forEach(function(ch){
+                        let r = ch.cb(msg);
+                        if (resp) resp = r; // If one of callback decided not to send response back, do not override it in global callbacks
+                    })
+                    
+                    // Cleanup Individual message channels to avoid memory leaks
+                    if (idx == 2) {
+                        delete proxy.ch[chanID]
+                    }
+                }
+            })
+
+            if (resp) {
+              process.stdout.write(`${resp.rawMeta.toString('hex')}${Buffer.from("\n").toString("hex")}${resp.http.toString('hex')}\n`)
+            }
+
+            return resp
+        }
+    }
+
+    // Clean up old messaged ID specific channels if they are older then 60s
+    let gc = function(gcTime){
+        let now = new Date();
+        for (k in proxy.ch) {
+            if (k.indexOf("#") == -1) continue;
+
+            proxy.ch[k] = proxy.ch[k].filter(function(ch){
+                return (now - ch.created) < gcTime
+            })
+
+            if (proxy.ch[k].length == 0) {
+                delete proxy.ch[k]
+            }
+        }
+    }
+    proxy.gc = gc
+
+    setInterval(function(){
+        gc(10 * 1000)
+    }, 1000);
+
+    const readline = require('readline');
+    const rl = readline.createInterface({
+          input: process.stdin
+    });
+
+    rl.on('line', function(line) {
+        let msg = parseMessage(line)
+        if (msg) {
+            proxy.emit(msg, line)
+        }
+    });
+
+    middleware = proxy;
+
+    return proxy;
+}
+
+
+function parseMessage(msg) {
+    try {
+        let payload = Buffer.from(msg, "hex");
+        let metaPos = payload.indexOf("\n");
+        let meta = payload.slice(0, metaPos);
+        let metaArr = meta.toString("ascii").split(" ");
+        let pType = metaArr[0];
+        let pID = metaArr[1];
+        let raw = payload.slice(metaPos + 1, payload.length);
+
+        return {
+            type: pType,
+            ID: pID,
+            rawMeta: meta,
+            meta: metaArr,
+            http: raw
+        }
+    } catch(e) {
+        fail(`Error while parsing incoming request: ${msg}`)
+    }
+}
+
+// Used to compare values from original and replayed responses
+// Accepts request id, regexp pattern for searching the compared value (should include capture group), and callback which returns both original and replayed matched value.
+// Example: 
+//
+//   // Compare HTTP headers for response and replayed response, and map values
+//   let tokMap = {};
+//
+//   gor.on("request", function(req) {
+//     let tok = gor.httpHeader(req.http, "Auth-Token");
+//     if (tok && tokMap[tok]) {
+//       req.http = gor.setHttpHeader(req.http, "Auth-Token", tokMap[tok]) 
+//     }
+//
+//     gor.searchResponses(req.ID, "X-Set-Token: (\w+)$", function(respTok, replTok) {
+//       tokMap[respTok] = replTok;
+//     })
+//
+//     return req;
+//   })
+//
+function searchResponses(id, searchPattern, callback) {
+    let re = new RegExp(searchPattern);
+
+    // Using regexp require converting buffer to string
+    // Before converting to string we can use initial `Buffer.indexOf` check
+    let indexPattern = searchPattern.split("(")[0];
+
+    if (!indexPattern) {
+        console.error("Search regexp should include capture group, pointing to the value: `prefix-(.*)`")
+        return
+    }
+
+    middleware.on("response", id, function(resp){
+        if (resp.http.indexOf(indexPattern) == -1) {
+            callback()
+            return resp
+        }
+
+        let respMatch = resp.http.toString('utf-8').match(re);
+        if (!respMatch) {
+            callback()
+            return resp
+        }
+
+        middleware.on("replay", id, function(repl) {
+            if (repl.http.indexOf(indexPattern) == -1) {
+                callback(respMatch[1]);
+                return repl;
+            }
+
+            let replMatch = repl.http.toString('utf-8').match(re);
+
+            if (!replMatch) {
+                callback(respMatch[1]);
+                return repl;
+            }
+        
+            callback(respMatch[1], replMatch[1]);
+            
+            return repl;
+        })
+
+        return resp;
+    })
+}
+
+
+// =========== HTTP parsing =================
+
+// Example HTTP payload record (including hidden characters):
+//
+//  POST / HTTP/1.1\r\n
+//  User-Agent: Node\r\n
+//  Content-Length: 5\r\n
+//  \r\n
+//  hello
+
+function httpMethod(payload) {
+    var pEnd = payload.indexOf(' ');
+    return payload.slice(0, pEnd).toString("ascii");
+}
+
+function httpPath(payload) {
+    var pStart = payload.indexOf(' ') + 1;
+    var pEnd = payload.indexOf(' ', pStart);
+    return payload.slice(pStart, pEnd).toString("ascii");
+}
+
+function setHttpPath(payload, newPath) {
+    var pStart = payload.indexOf(' ') + 1;
+    var pEnd = payload.indexOf(' ', pStart);
+
+    return Buffer.concat([payload.slice(0, pStart), Buffer.from(newPath), payload.slice(pEnd, payload.length)])
+}
+
+function httpPathParam(payload, name) {
+    let path = httpPath(payload);
+    let re = new RegExp(name + "=([^&$]+)");
+    let match = path.match(re);
+
+    if (match) return decodeURI(match[1]);
+}
+
+function setHttpPathParam(payload, name, value) {
+    let path = httpPath(payload);
+    let re = new RegExp(name + "=([^&$]+)");
+    let newPath = path.replace(re, name + "=" + encodeURI(value));
+    
+    // If we should add new param instead
+    if (newPath == path) {
+        if (newPath.indexOf("?") == -1) {
+            newPath += "?"
+        } else {
+            newPath += "&"
+        }
+
+        newPath += name + "=" + encodeURI(value);
+    }
+
+    return setHttpPath(payload, newPath)
+}
+
+// HTTP response have status code in same position as `path` for requests
+function httpStatus(payload) {
+    return httpPath(payload);
+}
+
+function setHttpStatus(payload, newStatus) {
+    return setHttpPath(payload, newStatus);
+}
+
+function httpHeaders(payload) {
+    var httpHeaderString = payload.slice(0,payload.indexOf("\r\n\r\n") + 4).toString().split("\n").slice(1);
+    var headers = {};
+
+    for (var item in httpHeaderString) {
+        var parts = httpHeaderString[item].split(":");
+
+        if (parts.length > 1) {
+            headers[parts[0]] = parts.slice(1).join(":").trim();    
+        }
+    }
+
+    return headers;
+}
+
+function httpHeader(payload, name) {
+    var currentLine = 0;
+    var i = 0;
+    var header = { start: -1, end: -1, valueStart: -1 }
+    var nameBuf = Buffer.from(name);
+    var nameBufLower = Buffer.from(name.toLowerCase());
+
+    while(c = payload[i]) {
+        if (c == 13) { // new line "\n"
+            currentLine++;
+            i++
+            header.end = i
+
+            if (currentLine > 0 && header.start > 0 && header.valueStart > 0) {
+                if (nameBuf.compare(payload, header.start, header.valueStart - 1) == 0 ||
+                    nameBufLower.compare(payload, header.start, header.valueStart - 1) == 0) { // ensure that headers are not case sensitive
+                    header.value = payload.slice(header.valueStart, header.end - 1).toString("utf-8").trim();
+                    header.name = payload.slice(header.start, header.valueStart - 1).toString("utf-8");
+                    return header
+                }
+            }
+
+            header.start = -1
+            header.valueStart = -1
+            continue;
+        } else if (c == 10) { // "\r"
+            i++
+            continue;
+        } else if (c == 58) { // ":" Header/value separator symbol
+            if (header.valueStart == -1) {
+                header.valueStart = i + 1;
+                i++
+                continue;
+            }
+        }
+
+        if (header.start == -1) header.start = i;
+
+        i++
+    }
+
+    return
+}
+
+function setHttpHeader(payload, name, value) {
+    let header = httpHeader(payload, name);
+    if (!header) {
+        let headerStart = payload.indexOf(13) + 1;
+        return Buffer.concat([payload.slice(0, headerStart + 1), Buffer.from(name + ": " + value + "\r\n"), payload.slice(headerStart + 1, payload.length)])
+    } else {
+        return Buffer.concat([payload.slice(0, header.valueStart), Buffer.from(" " + value + "\r\n"), payload.slice(header.end + 1, payload.length)])
+    }
+}
+
+function deleteHttpHeader(payload, name) {
+	let header = httpHeader(payload, name);
+
+    if (header) {
+        return Buffer.concat([payload.slice(0, header.start), payload.slice(header.end+1, payload.length)])
+    }
+
+	return payload
+}
+
+function httpBody(payload) {
+    return payload.slice(payload.indexOf("\r\n\r\n") + 4, payload.length);
+}
+
+function setHttpBody(payload, newBody) {
+    let p = setHttpHeader(payload, "Content-Length", newBody.length)
+    let headerEnd = p.indexOf("\r\n\r\n") + 4;
+    return Buffer.concat([p.slice(0, headerEnd), newBody])
+}
+
+function httpBodyParam(payload, name) {
+    let body = httpBody(payload);
+    let re = new RegExp(name + "=([^&$]+)");
+    if (body.indexOf(name + "=") != -1) {
+        let param = body.toString('utf-8').match(re);
+        if (param) {
+            return decodeURI(param[1]);
+        }
+    }
+}
+
+function setHttpBodyParam(payload, name, value) {
+    let body = httpBody(payload);
+    let re = new RegExp(name + "=([^&$]+)");
+
+    let newBody = body.toString('utf-8');
+
+    if (newBody.indexOf(name + "=") != -1 ) {
+        newBody = newBody.replace(re, name + "=" + encodeURI(value));
+    } else {
+        if (newBody.indexOf("=") != -1) {
+            newBody += "&";
+        }
+        newBody += name + "=" + value;
+    }
+    
+    return setHttpBody(payload, Buffer.from(newBody));
+}
+
+function setHttpCookie(payload, name, value) {
+    let h = httpHeader(payload, "Cookie");
+    let cookie = h ? h.value : "";
+    let cookies = cookie.split("; ").filter(function(v){ return v.indexOf(name + "=") != 0 })
+    cookies.push(name + "=" + value)
+    return setHttpHeader(payload, "Cookie", cookies.join("; "))
+}
+
+function httpCookie(payload, name) {
+    let h = httpHeader(payload, "Cookie");
+    let cookie = h ? h.value : "";
+    let value;
+    let cookies = cookie.split("; ").forEach(function(v){
+        if (v.indexOf(name + "=") == 0) {
+            value = v.substr(name.length + 1);
+        }
+    })
+    return value;
+}
+
+module.exports = {
+    init: init,
+    on: function(){ return middleware.on.apply(this, arguments) },
+    parseMessage: parseMessage,
+    searchResponses: searchResponses,
+    httpPath: httpPath,
+    httpMethod: httpMethod,
+    setHttpPath: setHttpPath,
+    httpPathParam: httpPathParam,
+    setHttpPathParam: setHttpPathParam,
+    httpStatus: httpStatus,
+    setHttpStatus: setHttpStatus,
+    httpHeader: httpHeader,
+    setHttpHeader: setHttpHeader,
+    deleteHttpHeader: deleteHttpHeader,
+    httpBody: httpBody,
+    setHttpBody: setHttpBody,
+    httpBodyParam: httpBodyParam,
+    setHttpBodyParam: setHttpBodyParam,
+    httpCookie: httpCookie,
+    setHttpCookie: setHttpCookie,
+    test: testRunner,
+    benchmark: testBenchmark,
+    httpHeaders: httpHeaders
+}
+
+
+// =========== Tests ==============
+
+function testRunner(){
+    ["init", "filter", "parseMessage", "httpMethod", "httpPath", "setHttpHeader", "deleteHttpHeader", "httpPathParam", "httpHeader", "httpBody", "setHttpBody", "httpBodyParam", "httpCookie", "setHttpCookie", "httpHeaders"].forEach(function(t){
+        console.log(`====== Start ${t} =======`)
+        eval(`TEST_${t}()`)
+        console.log(`====== End ${t} =======`)
+    })
+}
+
+function testBenchmark(){
+    const child_process = require('child_process');
+
+    let gor = init();
+    gor.on("message", function(){
+    });
+
+    gor.on("request", function(){
+    });
+
+    for (var i = 0; i<256; i++) {
+        let req = parseMessage(Buffer.from("1 2 3\nGET / HTTP/1.1\r\n\r\n").toString('hex'));
+        req.ID = +Date.now()
+        gor.emit(req);
+
+        gor.on("request", req.ID+"", function(){
+            gor.on("response", req.ID+"", function(){
+            })
+        })
+
+        if ( i % 3 == 0 ) {
+            let resp = parseMessage(Buffer.from("2 2 3\nHTTP/1.1 200 OK\r\n\r\n").toString('hex'));
+            resp.ID = req.ID
+            gor.emit(resp);
+        }
+    }
+    
+    child_process.execSync("sleep 0.01");
+
+    gor.gc(1) 
+    
+    fail(JSON.stringify(gor.ch))
+}
+
+// Just print in red color
+function fail(message) {
+    console.error("\x1b[31m[MIDDLEWARE] %s\x1b[0m", message)
+}
+
+function log(message) {
+    console.error(message)
+}
+
+function TEST_init() {
+    const child_process = require('child_process');
+
+    let received = 0;
+    let gor = init();
+    gor.on("message", function(){
+        received++; // should be called 3 times for for every request
+    });
+
+    gor.on("request", function(){
+        received++; // should be called 1 time only for request
+    });
+
+    gor.on("response", "2", function(){
+        received++; // should be called 1 time only for specific response
+    })
+
+    if (Object.keys(gor.ch).length != 3) {
+        return fail("Should create 3 channels");
+    }
+
+    let req = parseMessage(Buffer.from("1 2 3\nGET / HTTP/1.1\r\n\r\n").toString('hex'));
+    let resp = parseMessage(Buffer.from("2 2 3\nHTTP/1.1 200 OK\r\n\r\n").toString('hex'));
+    let resp2 = parseMessage(Buffer.from("2 3 3\nHTTP/1.1 200 OK\r\n\r\n").toString('hex'));
+    
+    gor.emit(req);
+    gor.emit(resp);
+    gor.emit(resp2);
+
+    child_process.execSync("sleep 0.01");
+
+    if (received != 5) {
+        fail(`Should receive 5 messages: ${received}`);
+    }
+}
+
+function TEST_filter() {
+    const child_process = require('child_process');
+
+    let gor = init();
+    gor.on("request", function(req){
+        if (httpPath(req.http) != "/filter") {
+            return req
+        }
+    });
+
+    gor.on("request", function(req){
+        return req
+    });
+
+
+    let reqPass = parseMessage(Buffer.from("1 2 3\nGET / HTTP/1.1\r\n\r\n").toString('hex'));
+    let reqFilter = parseMessage(Buffer.from("1 2 3\nGET /filter HTTP/1.1\r\n\r\n").toString('hex'));
+    
+    if (!gor.emit(reqPass)) {
+        return fail("Should not filter request")
+    }
+
+    if (gor.emit(reqFilter)) {
+        return fail("Should filter request even if one middleware rejected it")
+    }
+
+}
+
+function TEST_parseMessage() {
+    const exampleMessage = Buffer.from("1 2 3\nGET / HTTP/1.1\r\n\r\n").toString('hex')
+    let msg = parseMessage(exampleMessage)
+    let expected = { type: '1', ID: '2', meta: ["1", "2", "3"], http: Buffer.from("GET / HTTP/1.1\r\n\r\n") }
+
+    Object.keys(expected).forEach(function(k){
+        if (msg[k].toString() != expected[k].toString()) {
+            fail(`${k}: '${expected[k]}' != '${msg[k]}'`)
+        }
+    })
+}
+
+function TEST_httpPath() {
+    const examplePayload = "GET /test HTTP/1.1\r\n\r\n";
+
+    let payload = Buffer.from(examplePayload);
+    let path = httpPath(payload);
+
+    if (path != "/test") {
+        return fail(`Path '${patj}' != '/test'`)
+    }
+
+    let newPayload = setHttpPath(payload, '/')
+    if (newPayload.toString() != "GET / HTTP/1.1\r\n\r\n") {
+        return fail(`Malformed payload '${newPayload}'`)
+    }
+
+    newPayload = setHttpPath(payload, '/bigger')
+    if (newPayload.toString() != "GET /bigger HTTP/1.1\r\n\r\n") {
+        return fail(`Malformed payload '${newPayload}'`)
+    }
+}
+
+function TEST_httpMethod() {
+    const examplePayload = "GET /test HTTP/1.1\r\n\r\n";
+
+    let payload = Buffer.from(examplePayload);
+    let method = httpMethod(payload);
+
+    if (method != "GET") {
+        return fail(`Path '${method}' != 'GET'`)
+    }
+}
+
+
+function TEST_httpPathParam() {
+    let p = Buffer.from("GET / HTTP/1.1\r\n\r\n");
+
+    if (httpPathParam(p, "test")) {
+        return fail("Should not found param")
+    }
+
+    p = setHttpPathParam(p, "test", "123");
+    if (httpPath(p) != "/?test=123") {
+        return fail("Should set first param: " + httpPath(p));
+    }
+
+    if (httpPathParam(p, "test") != "123") {
+        return fail("Should get first param: " + httpPathParam(p, "test"));
+    }
+
+    p = setHttpPathParam(p, "qwer", "ty");
+    if (httpPath(p) != "/?test=123&qwer=ty") {
+        return fail("Should set second param: " + httpPath(p));
+    }
+
+    p = setHttpPathParam(p, "test", "4321");
+    if (httpPath(p) != "/?test=4321&qwer=ty") {
+        return fail("Should update first param: " + httpPath(p));
+    }
+
+    if (httpPathParam(p, "test") != "4321") {
+        return fail("Should update first param: " + httpPath(p));
+    }
+}
+
+function TEST_httpBodyParam() {
+    let p = Buffer.from("POST / HTTP/1.1\r\n\r\n");
+
+    if (httpBodyParam(p, "test")) {
+        return fail("Should not found param")
+    }
+
+    p = setHttpBodyParam(p, "test", "123");
+    if (httpBody(p).toString() != "test=123") {
+        return fail("Should set first param: " + httpBody(p).toString());
+    }
+
+    if (httpBodyParam(p, "test") != "123") {
+        return fail("Should get first param: " + httpBodyParam(p, "test"));
+    }
+
+    p = setHttpBodyParam(p, "qwer", "ty");
+    if (httpBody(p).toString() != "test=123&qwer=ty") {
+        return fail("Should set second param: " + httpBody(p).toString());
+    }
+
+    p = setHttpBodyParam(p, "test", "4321");
+    if (httpBody(p).toString() != "test=4321&qwer=ty") {
+        return fail("Should update first param: " + httpBody(p).toString());
+    }
+
+    if (httpBodyParam(p, "test") != "4321") {
+        return fail("Should update first param: " + httpBody(p).toString());
+    }
+}
+
+function TEST_httpHeader() {
+    const examplePayload = "GET / HTTP/1.1\r\nHost: localhost:3000\r\nUser-Agent: Node\r\nContent-Length:5\r\n\r\nhello";
+
+    let expected = {"Host": "localhost:3000", "User-Agent": "Node", "Content-Length": "5"}
+
+    Object.keys(expected).forEach(function(name){
+        let payload = Buffer.from(examplePayload);
+        let header = httpHeader(payload, name);
+        if (!header) {
+            fail(`Header not found. Was looking for: ${name}`)
+        }
+        if (header && header.value != expected[name]) {
+            fail(`${name}: '${expected[name]}' != '${header.value}'`)
+        }
+    })
+}
+
+
+function TEST_setHttpHeader() {
+    const examplePayload = "GET / HTTP/1.1\r\nUser-Agent: Node\r\nContent-Length: 5\r\n\r\nhello";
+
+    // Modify existing header
+    ["", "1", "Long test header"].forEach(function(ua){
+        let expected = `GET / HTTP/1.1\r\nUser-Agent: ${ua}\r\nContent-Length: 5\r\n\r\nhello`;
+        let p = Buffer.from(examplePayload);
+        p = setHttpHeader(p, "User-Agent", ua);
+        if (p != expected) {
+            console.error(`setHeader failed, expected User-Agent value: ${ua}.\n${p}`)
+        }
+    })
+
+    // Adding new header
+    let expected = `GET / HTTP/1.1\r\nX-Test: test\r\nUser-Agent: Node\r\nContent-Length: 5\r\n\r\nhello`;
+    let p = Buffer.from(examplePayload);
+    p = setHttpHeader(p, "X-Test", "test");
+    if (p != expected) {
+        console.error(`setHeader failed, expected new header 'X-Test' header: ${p}`)
+    }
+}
+
+function TEST_deleteHttpHeader() {
+    const examplePayload = "GET / HTTP/1.1\r\nUser-Agent: Node\r\nContent-Length: 5\r\n\r\nhello";
+
+    // Adding new header
+    let expected = `GET / HTTP/1.1\r\nContent-Length: 5\r\n\r\nhello`;
+    let p = Buffer.from(examplePayload);
+    p = deleteHttpHeader(p, "User-Agent", "test");
+    if (p != expected) {
+        console.error(`setHeader failed, expected delete header 'User-Agent' header: ${p}`)
+    }
+}
+
+function TEST_httpBody() {
+    const examplePayload = "GET / HTTP/1.1\r\nUser-Agent: Node\r\nContent-Length: 5\r\n\r\nhello";
+    let body = httpBody(Buffer.from(examplePayload));
+    if (body != "hello") {
+        fail(`'${body}' != 'hello'`)
+    }
+}
+
+function TEST_setHttpBody() {
+    const examplePayload = "GET / HTTP/1.1\r\nUser-Agent: Node\r\nContent-Length: 5\r\n\r\nhello";
+    let p = setHttpBody(Buffer.from(examplePayload), Buffer.from("hello, world!"));
+
+    if (p != "GET / HTTP/1.1\r\nUser-Agent: Node\r\nContent-Length: 13\r\n\r\nhello, world!") {
+        fail(`Wrong body: '${p}'`)
+    }
+}
+
+function TEST_httpCookie() {
+    const examplePayload = "GET / HTTP/1.1\r\nCookie: a=b; test=zxc\r\n\r\n";
+    let c = httpCookie(Buffer.from(examplePayload), "test");
+    if (c != "zxc") {
+        return fail(`Should get cookie: ${c}`);
+    }
+
+    c = httpCookie(Buffer.from(examplePayload), "nope");
+    if (c != null) {
+        return fail(`Should not find cookie: ${c}`);
+    }
+}
+
+function TEST_setHttpCookie() {
+    const examplePayload = "GET / HTTP/1.1\r\nCookie: a=b; test=zxc\r\n\r\n";
+    let p = setHttpCookie(Buffer.from(examplePayload), "test", "1");
+    if (p != "GET / HTTP/1.1\r\nCookie: a=b; test=1\r\n\r\n") {
+        return fail(`Should update cookie: ${p}`)
+    }
+
+    p = setHttpCookie(Buffer.from(examplePayload), "new", "one");
+    if (p != "GET / HTTP/1.1\r\nCookie: a=b; test=zxc; new=one\r\n\r\n") {
+        return fail(`Should add new cookie: ${p}`)
+    }
+}
+
+function TEST_httpHeaders() {
+    const examplePayload = "GET / HTTP/1.1\r\nHost: localhost:3000\r\nUser-Agent: Node\r\nContent-Length:5\r\n\r\nhello";
+
+    let expectedHeaders = {"Host": "localhost:3000", "User-Agent": "Node", "Content-Length": "5"}
+    let payload = Buffer.from(examplePayload);
+    let headers = httpHeaders(payload);
+
+    ["Host", "User-Agent", "Content-Length"].forEach(function(header){
+        let actual = headers[header];
+        let expected = expectedHeaders[header];
+
+        if (!actual) {
+            fail(`${header} Header was not found`);
+        }
+
+        if (actual != expected) {
+            fail(`${header} Header not Equal to Expected: ${expected} was ${actual}`);
+        }
+
+    })
+}
@@ -0,0 +1,16 @@
+{
+  "name": "goreplay_middleware",
+  "version": "0.1.19",
+  "description": "Package for writing middleware for GoReplay https://goreplay.org",
+  "main": "middleware.js",
+  "scripts": {
+    "test": "node -e \"var gor = require('./middleware.js'); gor.test(); process.exit()\"",
+    "benchmark": "node -e \"var gor = require('./middleware.js'); gor.benchmark(); process.exit()\""
+  },
+  "keywords": [
+    "middleware",
+    "goreplay"
+  ],
+  "author": "Leonid Bugaev",
+  "license": "LGPL-3.0"
+}
@@ -4,7 +4,6 @@ import (
 	"bytes"
 	"crypto/rand"
 	"encoding/hex"
-	"github.com/buger/gor/proto"
 	"io"
 	"net/http"
 	"net/http/httptest"
@@ -12,6 +11,8 @@ import (
 	"sync"
 	"testing"
 	"time"
+
+	"github.com/buger/goreplay/proto"
 )

 type fakeServiceCb func(string, int, []byte)
@@ -117,7 +118,7 @@ func TestEchoMiddleware(t *testing.T) {

 	// Catch traffic from one service
 	fromAddr := strings.Replace(from.Listener.Addr().String(), "[::]", "127.0.0.1", -1)
-	input := NewRAWInput(fromAddr, EnginePcap, true, testRawExpire, "")
+	input := NewRAWInput(fromAddr, EnginePcap, true, testRawExpire, "", "", "", 0)
 	defer input.Close()

 	// And redirect to another
@@ -179,7 +180,7 @@ func TestTokenMiddleware(t *testing.T) {

 	fromAddr := strings.Replace(from.Listener.Addr().String(), "[::]", "127.0.0.1", -1)
 	// Catch traffic from one service
-	input := NewRAWInput(fromAddr, EnginePcap, true, testRawExpire, "")
+	input := NewRAWInput(fromAddr, EnginePcap, true, testRawExpire, "", "", "", 0)
 	defer input.Close()

 	// And redirect to another
@@ -3,6 +3,7 @@ package main
 import (
 	"bufio"
 	"compress/gzip"
+	"errors"
 	"fmt"
 	"io"
 	"log"
@@ -24,14 +25,16 @@ var dateFileNameFuncs = map[string]func(*FileOutput) string{
 	"%M":  func(o *FileOutput) string { return time.Now().Format("04") },
 	"%S":  func(o *FileOutput) string { return time.Now().Format("05") },
 	"%NS": func(o *FileOutput) string { return fmt.Sprint(time.Now().Nanosecond()) },
-	"%r":  func(o *FileOutput) string { return o.currentID },
+	"%r":  func(o *FileOutput) string { return string(o.currentID) },
+	"%t":  func(o *FileOutput) string { return string(o.payloadType) },
 }

 type FileOutputConfig struct {
-	flushInterval time.Duration
-	sizeLimit     unitSizeVar
-	queueLimit    int
-	append        bool
+	flushInterval     time.Duration
+	sizeLimit         unitSizeVar
+	outputFileMaxSize unitSizeVar
+	queueLimit        int
+	append            bool
 }

 // FileOutput output plugin
@@ -44,7 +47,10 @@ type FileOutput struct {
 	chunkSize      int
 	writer         io.Writer
 	requestPerFile bool
-	currentID      string
+	currentID      []byte
+	payloadType    []byte
+	closed         bool
+	totalFileSize  int64

 	config *FileOutputConfig
 }
@@ -63,6 +69,9 @@ func NewFileOutput(pathTemplate string, config *FileOutputConfig) *FileOutput {
 	go func() {
 		for {
 			time.Sleep(config.flushInterval)
+			if o.closed {
+				break
+			}
 			o.updateName()
 			o.flush()
 		}
@@ -176,13 +185,12 @@ func (o *FileOutput) updateName() {

 func (o *FileOutput) Write(data []byte) (n int, err error) {
 	if o.requestPerFile {
-		o.currentID = string(payloadMeta(data)[1])
-		o.updateName()
+		meta := payloadMeta(data)
+		o.currentID = meta[1]
+		o.payloadType = meta[0]
 	}

-	if !isOriginPayload(data) {
-		return len(data), nil
-	}
+	o.updateName()

 	if o.file == nil || o.currentName != o.file.Name() {
 		o.mu.Lock()
@@ -208,8 +216,13 @@ func (o *FileOutput) Write(data []byte) (n int, err error) {
 	o.writer.Write(data)
 	o.writer.Write([]byte(payloadSeparator))

+	o.totalFileSize += int64(len(data) + len(payloadSeparator))
 	o.queueLength++

+	if Settings.outputFileConfig.outputFileMaxSize > 0 && o.totalFileSize >= int64(Settings.outputFileConfig.outputFileMaxSize) {
+		return len(data), errors.New("File output reached size limit")
+	}
+
 	return len(data), nil
 }

@@ -231,9 +244,11 @@ func (o *FileOutput) flush() {
 			o.writer.(*bufio.Writer).Flush()
 		}

-		if stat, err := o.file.Stat(); err != nil {
+		if stat, err := o.file.Stat(); err == nil {
 			o.chunkSize = int(stat.Size())
-		}
+		} else {
+            log.Println("Error accessing file sats", err)
+        }
 	}
 }

@@ -250,5 +265,7 @@ func (o *FileOutput) Close() error {
 		}
 		o.file.Close()
 	}
+
+	o.closed = true
 	return nil
 }
@@ -20,11 +20,12 @@ func parseDataUnit(s string) int64 {
 	if unit, ok := dataUnitMap[s[len(s)-1]]; ok {
 		size, _ := strconv.ParseInt(s[:len(s)-1], 10, 64)
 		return unit * size
-	} else {
-		// If no unit specified use bytes
-		size, _ := strconv.ParseInt(s, 10, 64)
-		return size
 	}
+
+	// If no unit specified use bytes
+	size, _ := strconv.ParseInt(s, 10, 64)
+	return size
+
 }

 type unitSizeVar int64
@@ -33,7 +34,7 @@ func (u unitSizeVar) String() string {
 	return strconv.Itoa(int(u))
 }

-func (u unitSizeVar) Set(s string) error {
-	u = unitSizeVar(parseDataUnit(s))
+func (u *unitSizeVar) Set(s string) error {
+	*u = unitSizeVar(parseDataUnit(s))
 	return nil
 }
@@ -65,9 +65,10 @@ func TestFileOutputWithNameCleaning(t *testing.T) {
 }

 func TestFileOutputPathTemplate(t *testing.T) {
-	output := &FileOutput{pathTemplate: "/tmp/log-%Y-%m-%d-%S", config: &FileOutputConfig{flushInterval: time.Minute, append: true}}
+	output := &FileOutput{pathTemplate: "/tmp/log-%Y-%m-%d-%S-%t", config: &FileOutputConfig{flushInterval: time.Minute, append: true}}
 	now := time.Now()
-	expectedPath := fmt.Sprintf("/tmp/log-%s-%s-%s-%s", now.Format("2006"), now.Format("01"), now.Format("02"), now.Format("05"))
+	output.payloadType = []byte("3")
+	expectedPath := fmt.Sprintf("/tmp/log-%s-%s-%s-%s-3", now.Format("2006"), now.Format("01"), now.Format("02"), now.Format("05"))
 	path := output.filename()

 	if expectedPath != path {
@@ -311,3 +312,37 @@ func TestFileOutputSort(t *testing.T) {
 		t.Error("Should properly sort file names using indexes", files, expected)
 	}
 }
+
+func TestFileOutputAppendSizeLimitOverflow(t *testing.T) {
+	rnd := rand.Int63()
+	name := fmt.Sprintf("/tmp/%d", rnd)
+
+	message := []byte("1 1 1\r\ntest")
+
+	messageSize := len(message) + len(payloadSeparator)
+
+	output := NewFileOutput(name, &FileOutputConfig{append: false, flushInterval: time.Minute, sizeLimit: unitSizeVar(2 * messageSize)})
+
+	output.Write([]byte("1 1 1\r\ntest"))
+	name1 := output.file.Name()
+
+	output.Write([]byte("1 1 1\r\ntest"))
+	name2 := output.file.Name()
+
+	output.flush()
+	output.updateName()
+
+	output.Write([]byte("1 1 1\r\ntest"))
+	name3 := output.file.Name()
+
+	if name2 != name1 || name1 != fmt.Sprintf("/tmp/%d_0", rnd) {
+		t.Error("Fast changes should happen in same file:", name1, name2, name3)
+	}
+
+	if name3 == name1 || name3 != fmt.Sprintf("/tmp/%d_1", rnd) {
+		t.Error("File name should change:", name1, name2, name3)
+	}
+
+	os.Remove(name1)
+	os.Remove(name3)
+}
@@ -2,10 +2,11 @@ package main

 import (
 	"io"
+	"log"
 	"sync/atomic"
 	"time"

-	"github.com/buger/gor/proto"
+	"github.com/buger/goreplay/proto"
 )

 const initialDynamicWorkers = 10
@@ -21,8 +22,12 @@ type response struct {
 type HTTPOutputConfig struct {
 	redirectLimit int

-	stats   bool
+	stats      bool
+	workersMin int
+	workersMax int
+	statsMs int
 	workers int
+	queueLen int

 	elasticSearch string

@@ -68,18 +73,18 @@ func NewHTTPOutput(address string, config *HTTPOutputConfig) io.Writer {
 	o.config = config

 	if o.config.stats {
-		o.queueStats = NewGorStat("output_http")
+		o.queueStats = NewGorStat("output_http", o.config.statsMs)
 	}

-	o.queue = make(chan []byte, 1000)
-	o.responses = make(chan response, 1000)
+	o.queue = make(chan []byte, o.config.queueLen)
+	o.responses = make(chan response, o.config.queueLen)
 	o.needWorker = make(chan int, 1)

 	// Initial workers count
-	if o.config.workers == 0 {
+	if o.config.workersMax == 0 {
 		o.needWorker <- initialDynamicWorkers
 	} else {
-		o.needWorker <- o.config.workers
+		o.needWorker <- o.config.workersMax
 	}

 	if o.config.elasticSearch != "" {
@@ -87,10 +92,6 @@ func NewHTTPOutput(address string, config *HTTPOutputConfig) io.Writer {
 		o.elasticSearch.Init(o.config.elasticSearch)
 	}

-	if len(Settings.middleware) > 0 {
-		o.config.TrackResponses = true
-	}
-
 	go o.workerMaster()

 	return o
@@ -102,11 +103,6 @@ func (o *HTTPOutput) workerMaster() {
 		for i := 0; i < newWorkers; i++ {
 			go o.startWorker()
 		}
-
-		// Disable dynamic scaling if workers poll fixed size
-		if o.config.workers != 0 {
-			return
-		}
 	}
 }

@@ -130,17 +126,16 @@ func (o *HTTPOutput) startWorker() {
 			deathCount = 0
 		case <-time.After(time.Millisecond * 100):
 			// When dynamic scaling enabled workers die after 2s of inactivity
-			if o.config.workers == 0 {
-				deathCount++
-			} else {
+			if o.config.workersMin == o.config.workersMax {
 				continue
 			}

+			deathCount++
 			if deathCount > 20 {
-				workersCount := atomic.LoadInt64(&o.activeWorkers)
+				workersCount := int(atomic.LoadInt64(&o.activeWorkers))

 				// At least 1 startWorker should be alive
-				if workersCount != 1 {
+				if workersCount != 1 && workersCount > o.config.workersMin {
 					atomic.AddInt64(&o.activeWorkers, -1)
 					return
 				}
@@ -163,11 +158,18 @@ func (o *HTTPOutput) Write(data []byte) (n int, err error) {
 		o.queueStats.Write(len(o.queue))
 	}

-	if o.config.workers == 0 {
-		workersCount := atomic.LoadInt64(&o.activeWorkers)
+	if o.config.workersMax != o.config.workersMin {
+		workersCount := int(atomic.LoadInt64(&o.activeWorkers))

-		if len(o.queue) > int(workersCount) {
-			o.needWorker <- len(o.queue)
+		if len(o.queue) > workersCount {
+			extraWorkersReq := len(o.queue) - workersCount + 1
+			maxWorkersAvailable := o.config.workersMax - workersCount
+			if extraWorkersReq > maxWorkersAvailable {
+				extraWorkersReq = maxWorkersAvailable
+			}
+			if extraWorkersReq > 0 {
+				o.needWorker <- extraWorkersReq
+			}
 		}
 	}

@@ -177,7 +179,9 @@ func (o *HTTPOutput) Write(data []byte) (n int, err error) {
 func (o *HTTPOutput) Read(data []byte) (int, error) {
 	resp := <-o.responses

-	Debug("[OUTPUT-HTTP] Received response:", string(resp.payload))
+	if Settings.debug {
+		Debug("[OUTPUT-HTTP] Received response:", string(resp.payload))
+	}

 	header := payloadHeader(ReplayedResponsePayload, resp.uuid, resp.roundTripTime, resp.startedAt)
 	copy(data[0:len(header)], header)
@@ -188,6 +192,11 @@ func (o *HTTPOutput) Read(data []byte) (int, error) {

 func (o *HTTPOutput) sendRequest(client *HTTPClient, request []byte) {
 	meta := payloadMeta(request)
+
+	if Settings.debug {
+		Debug(meta)
+	}
+
 	if len(meta) < 2 {
 		return
 	}
@@ -203,6 +212,7 @@ func (o *HTTPOutput) sendRequest(client *HTTPClient, request []byte) {
 	stop := time.Now()

 	if err != nil {
+		log.Println("Error when sending ", err, time.Now())
 		Debug("Request error:", err)
 	}

@@ -43,15 +43,19 @@ func TestHTTPOutput(t *testing.T) {
 	methods := HTTPMethods{[]byte("GET"), []byte("PUT"), []byte("POST")}
 	Settings.modifierConfig = HTTPModifierConfig{headers: headers, methods: methods}

-	output := NewHTTPOutput(server.URL, &HTTPOutputConfig{Debug: true})
+	http_output := NewHTTPOutput(server.URL, &HTTPOutputConfig{Debug: true, TrackResponses: true})
+	output := NewTestOutput(func(data []byte) {
+		wg.Done()
+	})

 	Plugins.Inputs = []io.Reader{input}
-	Plugins.Outputs = []io.Writer{output}
+	Plugins.Outputs = []io.Writer{http_output, output}

 	go Start(quit)

-	for i := 0; i < 100; i++ {
-		wg.Add(2) // OPTIONS should be ignored
+	for i := 0; i < 1; i++ {
+		// 2 http-output, 2 - test output request, 2 - test output http response
+		wg.Add(6) // OPTIONS should be ignored
 		input.EmitPOST()
 		input.EmitOPTIONS()
 		input.EmitGET()
@@ -2,13 +2,15 @@ package main

 import (
 	"encoding/json"
-	"github.com/Shopify/sarama"
-	"github.com/Shopify/sarama/mocks"
-	"github.com/buger/gor/proto"
 	"io"
 	"log"
 	"strings"
 	"time"
+
+	"github.com/buger/goreplay/proto"
+
+	"github.com/Shopify/sarama"
+	"github.com/Shopify/sarama/mocks"
 )

 // KafkaOutput is used for sending payloads to kafka in JSON format.
@@ -26,7 +28,7 @@ func NewKafkaOutput(address string, config *KafkaConfig) io.Writer {

 	var producer sarama.AsyncProducer

-	if config.producer.(*mocks.AsyncProducer) != nil {
+	if mock, ok := config.producer.(*mocks.AsyncProducer); ok && mock != nil {
 		producer = config.producer
 	} else {
 		c.Producer.RequiredAcks = sarama.WaitForLocal
@@ -1,9 +1,10 @@
 package main

 import (
+	"testing"
+
 	"github.com/Shopify/sarama"
 	"github.com/Shopify/sarama/mocks"
-	"testing"
 )

 func TestOutputKafkaRAW(t *testing.T) {
@@ -1,6 +1,7 @@
 package main

 import (
+	"crypto/tls"
 	"fmt"
 	"io"
 	"log"
@@ -16,18 +17,24 @@ type TCPOutput struct {
 	limit    int
 	buf      chan []byte
 	bufStats *GorStat
+	config   *TCPOutputConfig
+}
+
+type TCPOutputConfig struct {
+	secure bool
 }

 // NewTCPOutput constructor for TCPOutput
 // Initialize 10 workers which hold keep-alive connection
-func NewTCPOutput(address string) io.Writer {
+func NewTCPOutput(address string, config *TCPOutputConfig) io.Writer {
 	o := new(TCPOutput)

 	o.address = address
+	o.config = config

-	o.buf = make(chan []byte, 100)
+	o.buf = make(chan []byte, 1000)
 	if Settings.outputTCPStats {
-		o.bufStats = NewGorStat("output_tcp")
+		o.bufStats = NewGorStat("output_tcp", 5000)
 	}

 	for i := 0; i < 10; i++ {
@@ -59,11 +66,13 @@ func (o *TCPOutput) worker() {
 	defer conn.Close()

 	for {
-		conn.Write(<-o.buf)
+		data := <-o.buf
+		conn.Write(data)
 		_, err := conn.Write([]byte(payloadSeparator))

 		if err != nil {
-			log.Println("Lost connection with aggregator instance, reconnecting")
+			log.Println("INFO: TCP output connection closed, reconnecting")
+			o.buf <- data
 			go o.worker()
 			break
 		}
@@ -89,7 +98,11 @@ func (o *TCPOutput) Write(data []byte) (n int, err error) {
 }

 func (o *TCPOutput) connect(address string) (conn net.Conn, err error) {
-	conn, err = net.Dial("tcp", address)
+	if o.config.secure {
+		conn, err = tls.Dial("tcp", address, &tls.Config{})
+	} else {
+		conn, err = net.Dial("tcp", address)
+	}

 	return
 }
@@ -17,7 +17,7 @@ func TestTCPOutput(t *testing.T) {
 		wg.Done()
 	})
 	input := NewTestInput()
-	output := NewTCPOutput(listener.Addr().String())
+	output := NewTCPOutput(listener.Addr().String(), &TCPOutputConfig{})

 	Plugins.Inputs = []io.Reader{input}
 	Plugins.Outputs = []io.Writer{output}
@@ -69,7 +69,7 @@ func BenchmarkTCPOutput(b *testing.B) {
 		wg.Done()
 	})
 	input := NewTestInput()
-	output := NewTCPOutput(listener.Addr().String())
+	output := NewTCPOutput(listener.Addr().String(), &TCPOutputConfig{})

 	Plugins.Inputs = []io.Reader{input}
 	Plugins.Outputs = []io.Writer{output}
@@ -5,7 +5,6 @@ import (
 	"reflect"
 	"strings"
 	"sync"
-	"time"
 )

 // InOutPlugins struct for holding references to plugins
@@ -107,15 +106,15 @@ func InitPlugins() {
 	}

 	for _, options := range Settings.inputRAW {
-		registerPlugin(NewRAWInput, options, engine, Settings.inputRAWTrackResponse, time.Duration(0), Settings.inputRAWRealIPHeader)
+		registerPlugin(NewRAWInput, options, engine, Settings.inputRAWTrackResponse, Settings.inputRAWExpire, Settings.inputRAWRealIPHeader, Settings.inputRAWBpfFilter, Settings.inputRAWTimestampType, Settings.inputRawBufferSize)
 	}

 	for _, options := range Settings.inputTCP {
-		registerPlugin(NewTCPInput, options)
+		registerPlugin(NewTCPInput, options, &Settings.inputTCPConfig)
 	}

 	for _, options := range Settings.outputTCP {
-		registerPlugin(NewTCPOutput, options)
+		registerPlugin(NewTCPOutput, options, &Settings.outputTCPConfig)
 	}

 	for _, options := range Settings.inputFile {
@@ -4,9 +4,9 @@ package proto

 func Fuzz(data []byte) int {

-    ParseHeaders([][]byte{data}, func(header []byte, value []byte) bool {
-      return true
-    })
+	ParseHeaders([][]byte{data}, func(header []byte, value []byte) bool {
+		return true
+	})

-    return 1
+	return 1
 }
@@ -18,7 +18,8 @@ package proto

 import (
 	"bytes"
-	"github.com/buger/gor/byteutils"
+
+	"github.com/buger/goreplay/byteutils"
 )

 // In HTTP newline defined by 2 bytes (for both windows and *nix support)
@@ -32,7 +33,7 @@ var HeaderDelim = []byte(": ")

 // MIMEHeadersEndPos finds end of the Headers section, which should end with empty line.
 func MIMEHeadersEndPos(payload []byte) int {
-	return bytes.Index(payload, EmptyLine)
+	return bytes.Index(payload, EmptyLine) + 4
 }

 // MIMEHeadersStartPos finds start of Headers section
@@ -41,82 +42,23 @@ func MIMEHeadersStartPos(payload []byte) int {
 	return bytes.Index(payload, CLRF) + 2 // Find first line end
 }

-func isLower(b byte) bool {
-	if 'a' <= b && b <= 'z' {
-		return true
-	}
-
-	return false
-}
-
-func toUpper(b byte) byte {
-	if 'a' <= b && b <= 'z' {
-		b -= 'a' - 'A'
-	}
-	return b
-}
-
-func toLower(b byte) byte {
-	if 'A' <= b && b <= 'Z' {
-		b += 'a' - 'A'
-	}
-	return b
-}
-
 func headerIndex(payload []byte, name []byte) int {
-	isLower := isLower(name[0])
 	i := 0
-
 	for {
-		if i >= len(payload) {
+		// we need enough space for at least '\n' and the header name
+		if i >= (len(payload) - len(name) - 1) {
 			return -1
 		}

 		if payload[i] == '\n' {
 			i++
-
-			// We are at the end
-			if i == len(payload) {
-				return -1
-			}
-
-			if payload[i] == name[0] ||
-				(!isLower && payload[i] == toLower(name[0])) ||
-				(isLower && payload[i] == toUpper(name[0])) {
-
-				i++
-				j := 1
-				for {
-					if j == len(name) {
-						// Matched, and return start of the header
-						return i - len(name)
-					}
-
-					if payload[i] != name[j] {
-						break
-					}
-
-					// If compound header name do one more case check: Content-Length or Transfer-Encoding
-					if name[j] == '-' {
-						i++
-						j++
-
-						if !(payload[i] == name[j] ||
-							(!isLower && payload[i] == toLower(name[j])) ||
-							(isLower && payload[i] == toUpper(name[j]))) {
-							break
-						}
-					}
-
-					j++
-					i++
-				}
+			if bytes.EqualFold(name, payload[i:i+len(name)]) {
+				return i
 			}
 		}
-
 		i++
-	}

+	}
 	return -1
 }

@@ -181,7 +123,7 @@ func HeadersEqual(h1 []byte, h2 []byte) bool {

 // Parsing headers from multiple payloads
 func ParseHeaders(payloads [][]byte, cb func(header []byte, value []byte) bool) {
-	hS := [2]int{0, 0} // header start
+	hS := [2]int{0, 0}   // header start
 	hE := [2]int{-1, -1} // header end
 	vS := [2]int{-1, -1} // value start
 	vE := [2]int{-1, -1} // value end
@@ -337,14 +279,35 @@ func DeleteHeader(payload, name []byte) []byte {
 // Body returns request/response body
 func Body(payload []byte) []byte {
 	// 4 -> len(EMPTY_LINE)
-	return payload[MIMEHeadersEndPos(payload)+4:]
+	if len(payload) < 4 {
+		return []byte{}
+	}
+
+	return payload[MIMEHeadersEndPos(payload):]
 }

 // Path takes payload and retuns request path: Split(firstLine, ' ')[1]
 func Path(payload []byte) []byte {
 	start := bytes.IndexByte(payload, ' ') + 1
+	eol := bytes.IndexByte(payload[start:], '\r')
 	end := bytes.IndexByte(payload[start:], ' ')

+	if eol > 0 {
+		if end == -1 || eol < end {
+			return payload[start : start+eol]
+		}
+	} else { // support for legacy clients
+		eol = bytes.IndexByte(payload[start:], '\n')
+
+		if eol > 0 && (end == -1 || eol < end) {
+			return payload[start : start+eol]
+		}
+	}
+
+	if end < 0 {
+		return payload[start:len(payload)]
+	}
+
 	return payload[start : start+end]
 }

@@ -449,7 +412,7 @@ func Status(payload []byte) []byte {
 }

 var httpMethods []string = []string{
-	"GET ", "OPTI", "HEAD", "POST", "PUT ", "DELE", "TRAC", "CONN", "PATC" /* custom methods */, "BAN", "PURG",
+	"GET ", "OPTI", "HEAD", "POST", "PUT ", "DELE", "TRAC", "CONN", "PATC" /* custom methods */, "BAN ", "PURG", "PROP", "MKCO", "COPY", "MOVE", "LOCK", "UNLO",
 }

 func IsHTTPPayload(payload []byte) bool {
@@ -65,13 +65,13 @@ func TestHeader(t *testing.T) {
 }

 func TestMIMEHeadersEndPos(t *testing.T) {
-	head := []byte("POST /post HTTP/1.1\r\nContent-Length: 7\r\nHost: www.w3.org")
+	head := []byte("POST /post HTTP/1.1\r\nContent-Length: 7\r\nHost: www.w3.org\r\n\r\n")
 	payload := []byte("POST /post HTTP/1.1\r\nContent-Length: 7\r\nHost: www.w3.org\r\n\r\na=1&b=2")

 	end := MIMEHeadersEndPos(payload)

 	if !bytes.Equal(payload[:end], head) {
-		t.Error("Wrong headers end position:", end)
+		t.Error("Wrong headers end position:", end, head, payload[:end])
 	}
 }

@@ -80,10 +80,10 @@ func TestMIMEHeadersStartPos(t *testing.T) {
 	payload := []byte("POST /post HTTP/1.1\r\nContent-Length: 7\r\nHost: www.w3.org\r\n\r\na=1&b=2")

 	start := MIMEHeadersStartPos(payload)
-	end := MIMEHeadersEndPos(payload)
+	end := MIMEHeadersEndPos(payload) - 4

 	if !bytes.Equal(payload[start:end], headers) {
-		t.Error("Wrong headers end position:", start, end)
+		t.Error("Wrong headers end position:", start, end, payload[start:end])
 	}
 }

@@ -153,8 +153,8 @@ func TestFuzzCrashers(t *testing.T) {

 	for _, f := range crashers {
 		ParseHeaders([][]byte{[]byte(f)}, func(header []byte, value []byte) bool {
-	      return true
-	    })
+			return true
+		})
 	}
 }

@@ -237,6 +237,24 @@ func TestPath(t *testing.T) {
 	if path = Path(payload); !bytes.Equal(path, []byte("/post")) {
 		t.Error("Should find path", string(path))
 	}
+
+	payload = []byte("GET /get\r\n\r\nHost: www.w3.org\r\n\r\n")
+
+	if path = Path(payload); !bytes.Equal(path, []byte("/get")) {
+		t.Error("Should find path", string(path))
+	}
+
+	payload = []byte("GET /get\n")
+
+	if path = Path(payload); !bytes.Equal(path, []byte("/get")) {
+		t.Error("Should find path", string(path))
+	}
+
+	payload = []byte("GET /get")
+
+	if path = Path(payload); !bytes.Equal(path, []byte("/get")) {
+		t.Error("Should find path", string(path))
+	}
 }

 func TestSetPath(t *testing.T) {
@@ -16,10 +16,6 @@ import (
 	"bytes"
 	"encoding/binary"
 	"fmt"
-	"github.com/buger/gor/proto"
-	"github.com/google/gopacket"
-	"github.com/google/gopacket/layers"
-	"github.com/google/gopacket/pcap"
 	"io"
 	"log"
 	"net"
@@ -29,14 +25,20 @@ import (
 	"strings"
 	"sync"
 	"time"
+
+	"github.com/buger/goreplay/proto"
+
+	"github.com/google/gopacket"
+	"github.com/google/gopacket/layers"
+	"github.com/google/gopacket/pcap"
 )

 var _ = fmt.Println

 type packet struct {
-	srcIP		[]byte
-	data		[]byte
-	timestamp	time.Time
+	srcIP     []byte
+	data      []byte
+	timestamp time.Time
 }

 // Listener handle traffic capture
@@ -70,6 +72,13 @@ type Listener struct {
 	trackResponse bool
 	messageExpire time.Duration

+	bpfFilter     string
+	timestampType string
+	overrideSnapLen bool
+	immediateMode bool
+
+	bufferSize int
+
 	conn        net.PacketConn
 	pcapHandles []*pcap.Handle

@@ -91,7 +100,7 @@ const (
 )

 // NewListener creates and initializes new Listener object
-func NewListener(addr string, port string, engine int, trackResponse bool, expire time.Duration) (l *Listener) {
+func NewListener(addr string, port string, engine int, trackResponse bool, expire time.Duration, bpfFilter string, timestampType string, bufferSize int, overrideSnapLen bool, immediateMode bool) (l *Listener) {
 	l = &Listener{}

 	l.packetsChan = make(chan *packet, 10000)
@@ -105,6 +114,11 @@ func NewListener(addr string, port string, engine int, trackResponse bool, expir
 	l.respAliases = make(map[uint32]*TCPMessage)
 	l.respWithoutReq = make(map[uint32]tcpID)
 	l.trackResponse = trackResponse
+	l.bpfFilter = bpfFilter
+	l.timestampType = timestampType
+	l.immediateMode = immediateMode
+	l.bufferSize = bufferSize
+	l.overrideSnapLen = overrideSnapLen

 	l.addr = addr
 	_port, _ := strconv.Atoi(port)
@@ -241,7 +255,7 @@ func (e *DeviceNotFoundError) Error() string {
 	devices, _ := pcap.FindAllDevs()

 	if len(devices) == 0 {
-		return "Can't get list of network interfaces, ensure that you running Gor as root user or sudo.\nTo run as non-root users see this docs https://github.com/buger/gor/wiki/Running-as-non-root-user"
+		return "Can't get list of network interfaces, ensure that you running Gor as root user or sudo.\nTo run as non-root users see this docs https://github.com/buger/goreplay/wiki/Running-as-non-root-user"
 	}

 	var msg string
@@ -303,9 +317,9 @@ func findPcapDevices(addr string) (interfaces []pcap.Interface, err error) {

 	if len(interfaces) == 0 {
 		return nil, &DeviceNotFoundError{addr}
-	} else {
-		return interfaces, nil
 	}
+
+	return interfaces, nil
 }

 func (t *Listener) readPcap() {
@@ -324,12 +338,45 @@ func (t *Listener) readPcap() {

 	for _, d := range devices {
 		go func(device pcap.Interface) {
-			handle, err := pcap.OpenLive(device.Name, 65536, true, t.messageExpire)
+			inactive, err := pcap.NewInactiveHandle(device.Name)
 			if err != nil {
 				log.Println("Pcap Error while opening device", device.Name, err)
 				wg.Done()
 				return
 			}
+
+			if t.timestampType != "" {
+				if tt, terr := pcap.TimestampSourceFromString(t.timestampType); terr != nil {
+					log.Println("Supported timestamp types: ", inactive.SupportedTimestamps(), device.Name)
+				} else if terr := inactive.SetTimestampSource(tt); terr != nil {
+					log.Println("Supported timestamp types: ", inactive.SupportedTimestamps(), device.Name)
+				}
+			}
+
+			if it, err := net.InterfaceByName(device.Name); err == nil && !t.overrideSnapLen {
+				// Auto-guess max length of packet to capture
+				inactive.SetSnapLen(it.MTU + 68*2)
+			} else {
+				inactive.SetSnapLen(65536)
+			}
+
+			inactive.SetTimeout(t.messageExpire)
+			inactive.SetPromisc(true)
+			inactive.SetImmediateMode(t.immediateMode)
+			if t.immediateMode {
+				log.Println("Setting immediate mode")
+			}
+			if t.bufferSize > 0 {
+				inactive.SetBufferSize(t.bufferSize)
+			}
+
+			handle, herr := inactive.Activate()
+			if herr != nil {
+				log.Println("PCAP Activate error:", herr)
+				wg.Done()
+				return
+			}
+
 			defer handle.Close()

 			t.mu.Lock()
@@ -368,6 +415,10 @@ func (t *Listener) readPcap() {
 					bpf = "tcp dst port " + strconv.Itoa(int(t.port)) + " and (" + bpfDstHost + ")"
 				}

+				if t.bpfFilter != "" {
+					bpf = t.bpfFilter
+				}
+
 				if err := handle.SetBPFFilter(bpf); err != nil {
 					log.Println("BPF filter error:", err, "Device:", device.Name, bpf)
 					wg.Done()
@@ -415,12 +466,12 @@ func (t *Listener) readPcap() {
 					of = 4
 				case layers.LinkTypeLoop:
 					of = 4
-				case layers.LinkTypeRaw:
+				case layers.LinkTypeRaw, layers.LayerTypeIPv4:
 					of = 0
 				case layers.LinkTypeLinuxSLL:
 					of = 16
 				default:
-					log.Println("Unknown packet layer", packet)
+					log.Println("Unknown packet layer", decoder, packet)
 					break
 				}

@@ -542,6 +593,13 @@ func (t *Listener) readPcapFile() {
 	if handle, err := pcap.OpenOffline(t.addr); err != nil {
 		log.Fatal(err)
 	} else {
+		if t.bpfFilter != "" {
+			if err := handle.SetBPFFilter(t.bpfFilter); err != nil {
+				log.Println("BPF filter error:", err)
+				return
+			}
+		}
+
 		t.readyCh <- true
 		packetSource := gopacket.NewPacketSource(handle, handle.LinkType())

@@ -560,12 +618,12 @@ func (t *Listener) readPcapFile() {
 				tcp, _ := tcpLayer.(*layers.TCP)
 				data = append(tcp.LayerContents(), tcp.LayerPayload()...)

-				if tcp.SrcPort >= 32768 && tcp.SrcPort <= 61000 {
-					copy(data[0:2], []byte{0, 0})
-					copy(data[2:4], []byte{0, 1})
+				if uint16(tcp.DstPort) == t.port {
+					copy(data[0:2], []byte{byte(tcp.SrcPort >> 8), byte(tcp.SrcPort)})
+					copy(data[2:4], []byte{byte(tcp.DstPort >> 8), byte(tcp.DstPort)})
 				} else {
-					copy(data[0:2], []byte{0, 1})
-					copy(data[2:4], []byte{0, 0})
+					copy(data[0:2], []byte{byte(tcp.DstPort >> 8), byte(tcp.DstPort)})
+					copy(data[2:4], []byte{byte(tcp.SrcPort >> 8), byte(tcp.SrcPort)})
 				}
 			} else {
 				continue
@@ -631,16 +689,10 @@ func (t *Listener) readRAWSocket() {
 }

 func (t *Listener) buildPacket(packetSrcIP []byte, packetData []byte, timestamp time.Time) *packet {
-	copyPacketSrcIP := make([]byte, 16)
-	copyPacketData := make([]byte, len(packetData))
-
-	copy(copyPacketSrcIP, packetSrcIP)
-	copy(copyPacketData, packetSrcIP)
-
 	return &packet{
-		srcIP: packetSrcIP,
-		data: packetData,
-		timestamp:timestamp,
+		srcIP:     packetSrcIP,
+		data:      packetData,
+		timestamp: timestamp,
 	}
 }

@@ -677,18 +729,30 @@ func (t *Listener) processTCPPacket(packet *TCPPacket) {
 		}
 	}()

+	var responseRequest *TCPMessage
 	var message *TCPMessage

 	isIncoming := packet.DestPort == t.port

+	if !isIncoming {
+		responseRequest, _ = t.respAliases[packet.Ack]
+	}
+
 	// Seek for 100-expect chunks
-	if parentAck, ok := t.seqWithData[packet.Seq]; ok {
+	// `packet.Ack != parentAck` is protection for clients who send data without ignoring server 100-continue response, e.g have data chunks have same Ack
+	if parentAck, ok := t.seqWithData[packet.Seq]; ok && packet.Ack != parentAck {
+		// Skip zero-length chunks https://github.com/buger/goreplay/issues/496
+		if len(packet.Data) == 0 {
+			return
+		}
+
 		// In case if non-first data chunks comes first
 		for _, m := range t.messages {
 			if m.Ack == packet.Ack && bytes.Equal(m.packets[0].Addr, packet.Addr) {
 				t.deleteMessage(m)

 				if m.AssocMessage != nil {
+					m.AssocMessage.setAssocMessage(nil)
 					m.setAssocMessage(nil)
 				}

@@ -717,12 +781,6 @@ func (t *Listener) processTCPPacket(packet *TCPPacket) {
 		packet.UpdateAck(alias)
 	}

-	var responseRequest *TCPMessage
-
-	if !isIncoming {
-		responseRequest, _ = t.respAliases[packet.Ack]
-	}
-
 	message, ok := t.messages[packet.ID]

 	if !ok {
@@ -744,8 +802,9 @@ func (t *Listener) processTCPPacket(packet *TCPPacket) {

 	// Handling Expect: 100-continue requests
 	if message.expectType == httpExpect100Continue && len(message.packets) == message.headerPacket+1 {
-		seq := packet.Seq + uint32(message.Size())
+		seq := packet.Seq + uint32(len(packet.Data))
 		t.seqWithData[seq] = packet.Ack
+
 		message.DataSeq = seq
 		message.complete = false

@@ -755,8 +814,9 @@ func (t *Listener) processTCPPacket(packet *TCPPacket) {
 				t.deleteMessage(m)
 				if m.AssocMessage != nil {
 					message.setAssocMessage(m.AssocMessage)
+					m.AssocMessage.setAssocMessage(nil)
 				}
-				// log.Println("2: Adding ack alias:", m.Ack, packet.Ack)
+
 				t.ackAliases[m.Ack] = packet.Ack

 				for _, pkt := range m.packets {
@@ -770,7 +830,13 @@ func (t *Listener) processTCPPacket(packet *TCPPacket) {
 		packet.Data = proto.DeleteHeader(packet.Data, bExpectHeader)
 	}

-	// log.Println("Received message:", string(message.Bytes()), message.ID(), t.messages)
+	// If client do sends Expect: 100-continue but do not respect server response
+	if message.expectType == httpExpect100Continue && (message.headerPacket != -1 && len(message.packets) > message.headerPacket+1) {
+		delete(t.seqWithData, message.DataSeq)
+		seq := packet.Seq + uint32(len(packet.Data))
+		t.seqWithData[seq] = packet.Ack
+		message.DataSeq = seq
+	}

 	if isIncoming {
 		// If message have multiple packets, delete previous alias
@@ -12,7 +12,7 @@ import (
 func TestRawListenerInput(t *testing.T) {
 	var req, resp *TCPMessage

-	listener := NewListener("", "0", EnginePcap, true, 10*time.Millisecond)
+	listener := NewListener("", "0", EnginePcap, true, 10*time.Millisecond, "", "", 0, false)
 	defer listener.Close()

 	reqPacket := buildPacket(true, 1, 1, []byte("GET / HTTP/1.1\r\n\r\n"), time.Now())
@@ -46,10 +46,148 @@ func TestRawListenerInput(t *testing.T) {
 	}
 }

+func firstPacket(payload []byte) *TCPPacket {
+	return buildPacket(
+		true,
+		1,
+		1,
+		payload,
+		time.Now(),
+	)
+}
+
+func nextPacket(prev *TCPPacket, payload []byte) *TCPPacket {
+	return buildPacket(
+		prev.SrcPort == 1,
+		prev.Ack,
+		prev.Seq+uint32(len(prev.Data)),
+		payload,
+		prev.timestamp.Add(time.Millisecond),
+	)
+}
+
+func responsePacket(prev *TCPPacket, payload []byte) *TCPPacket {
+	return buildPacket(
+		!(prev.SrcPort == 1),
+		prev.Seq+uint32(len(prev.Data)),
+		prev.Ack,
+		payload,
+		prev.timestamp.Add(time.Millisecond),
+	)
+}
+
+func TestHEADRequestNoBody(t *testing.T) {
+	listener := NewListener("", "0", EnginePcap, true, 10*time.Millisecond, "", "", 0, false)
+	defer listener.Close()
+
+	reqPacket := firstPacket([]byte("HEAD / HTTP/1.1\r\nContent-Length: 0\r\n\r\n"))
+	respPacket := responsePacket(reqPacket, []byte("HTTP/1.1 200 OK\r\nContent-Length: 100\r\n\r\n"))
+
+	listener.packetsChan <- reqPacket.dump()
+	listener.packetsChan <- respPacket.dump()
+
+	var req, resp *TCPMessage
+	select {
+	case req = <-listener.messagesChan:
+	case <-time.After(time.Millisecond):
+		t.Error("Should return request immediately")
+		return
+	}
+
+	if !req.IsIncoming {
+		t.Error("Should be request")
+	}
+
+	select {
+	case resp = <-listener.messagesChan:
+	case <-time.After(20 * time.Millisecond):
+		t.Error("Should return response immediately")
+		return
+	}
+
+	if resp.IsIncoming {
+		t.Error("Should be response")
+	}
+}
+
+func TestSingleAck100Continue(t *testing.T) {
+	listener := NewListener("", "0", EnginePcap, true, 10*time.Millisecond, "", "", 0, false)
+	defer listener.Close()
+
+	reqPacket1 := firstPacket([]byte("POST / HTTP/1.1\r\nExpect: 100-continue\r\nContent-Length: 4\r\n\r\n"))
+	respPacket1 := responsePacket(reqPacket1, []byte(""))
+	respPacket2 := responsePacket(reqPacket1, []byte("HTTP/1.1 100 Continue\r\n"))
+	reqPacket2 := responsePacket(respPacket2, []byte("DATA"))
+	respPacket3 := responsePacket(reqPacket2, []byte("HTTP/1.1 200 OK\r\n\r\n"))
+
+	result := []byte("POST / HTTP/1.1\r\nContent-Length: 4\r\n\r\nDATA")
+
+	testRawListener100Continue(t, listener, result,
+		reqPacket1,
+		respPacket1, respPacket2,
+		reqPacket2,
+		respPacket3)
+}
+
+func Test100ContinueWithoutWaiting(t *testing.T) {
+	listener := NewListener("", "0", EnginePcap, true, 10*time.Millisecond, "", "", 0, false)
+	defer listener.Close()
+
+	req1 := firstPacket([]byte("POST / HTTP/1.1\r\nExpect: 100-continue\r\nContent-Length: 4\r\n\r\n"))
+	req2 := nextPacket(req1, []byte("DATA"))
+	resp1 := responsePacket(req1, []byte("HTTP/1.1 100 Continue\r\n"))
+	resp2 := responsePacket(req2, []byte("HTTP/1.1 200 OK\r\n\r\n"))
+
+	result := []byte("POST / HTTP/1.1\r\nContent-Length: 4\r\n\r\nDATA")
+
+	testRawListener100Continue(t, listener, result,
+		req1, req2, resp1, resp2)
+}
+
+// Client first sends data without waiting 100-continue, but once response received, generate packets based on Ack payload
+func Test100ContinueMixed(t *testing.T) {
+	listener := NewListener("", "0", EnginePcap, true, 10*time.Millisecond, "", "", 0, false)
+	defer listener.Close()
+
+	req1 := firstPacket([]byte("POST / HTTP/1.1\r\nExpect: 100-continue\r\nContent-Length: 12\r\n\r\n"))
+	req2 := nextPacket(req1, []byte("DAT1"))
+	resp1 := responsePacket(req1, []byte("HTTP/1.1 100 Continue\r\n\r\n"))
+	req3 := responsePacket(resp1, []byte("DAT2"))
+	req3.Seq = req2.Seq + uint32(len(req2.Data))
+	req4 := nextPacket(req3, []byte("DAT3"))
+	resp2 := responsePacket(req4, []byte("HTTP/1.1 200 OK\r\n\r\n"))
+
+	result := []byte("POST / HTTP/1.1\r\nContent-Length: 12\r\n\r\nDAT1DAT2DAT3")
+
+	testRawListener100Continue(t, listener, result,
+		req1, req2, req3, req4, resp1, resp2)
+}
+
+func TestDoubleAck100Continue(t *testing.T) {
+	listener := NewListener("", "0", EnginePcap, true, 10*time.Millisecond, "", "", 0, false)
+	defer listener.Close()
+
+	reqPacket1 := firstPacket([]byte("POST / HTTP/1.1\r\nExpect: 100-continue\r\nContent-Length: 4\r\n\r\n"))
+
+	respPacket1 := responsePacket(reqPacket1, []byte(""))
+	respPacket2 := responsePacket(reqPacket1, []byte("HTTP/1.1 100 Continue\r\n"))
+	reqPacket2 := responsePacket(respPacket2, []byte(""))
+	reqPacket3 := responsePacket(respPacket2, []byte("DATA"))
+	respPacket3 := responsePacket(reqPacket3, []byte("HTTP/1.1 200 OK\r\n\r\n"))
+
+	result := []byte("POST / HTTP/1.1\r\nContent-Length: 4\r\n\r\nDATA")
+
+	testRawListener100Continue(t, listener, result,
+		reqPacket1,
+		respPacket1, respPacket2,
+		reqPacket2, reqPacket3,
+		respPacket3)
+}
+
 func TestRawListenerInputResponseByClose(t *testing.T) {
 	var req, resp *TCPMessage

-	listener := NewListener("", "0", EnginePcap, true, 10*time.Millisecond)
+	listener := NewListener("", "0", EnginePcap, true, 10*time.Millisecond, "", "", 0, false)
 	defer listener.Close()

 	reqPacket := buildPacket(true, 1, 1, []byte("GET / HTTP/1.1\r\n\r\n"), time.Now())
@@ -89,7 +227,7 @@ func TestRawListenerInputResponseByClose(t *testing.T) {
 func TestRawListenerInputWithoutResponse(t *testing.T) {
 	var req *TCPMessage

-	listener := NewListener("", "0", EnginePcap, false, 10*time.Millisecond)
+	listener := NewListener("", "0", EnginePcap, false, 10*time.Millisecond, "", "", 0, false)
 	defer listener.Close()

 	reqPacket := buildPacket(true, 1, 1, []byte("GET / HTTP/1.1\r\n\r\n"), time.Now())
@@ -111,11 +249,11 @@ func TestRawListenerInputWithoutResponse(t *testing.T) {
 func TestRawListenerResponse(t *testing.T) {
 	var req, resp *TCPMessage

-	listener := NewListener("", "0", EnginePcap, true, 10*time.Millisecond)
+	listener := NewListener("", "0", EnginePcap, true, 10*time.Millisecond, "", "", 0, false)
 	defer listener.Close()

-	reqPacket := buildPacket(true, 1, 1, []byte("GET / HTTP/1.1\r\n\r\n"), time.Now())
-	respPacket := buildPacket(false, 1+uint32(len(reqPacket.Data)), 2, []byte("HTTP/1.1 200 OK\r\n\r\n"), time.Now())
+	reqPacket := firstPacket([]byte("GET / HTTP/1.1\r\n\r\n"))
+	respPacket := responsePacket(reqPacket, []byte("HTTP/1.1 200 OK\r\n\r\n"))

 	// If response packet comes before request
 	listener.packetsChan <- respPacket.dump()
@@ -148,62 +286,37 @@ func TestRawListenerResponse(t *testing.T) {
 	}
 }

+func get100ContinuePackets() (req []*TCPPacket, resp []*TCPPacket) {
+	req1 := firstPacket([]byte("POST / HTTP/1.1\r\nExpect: 100-continue\r\nContent-Length: 2\r\n\r\n"))
+	resp1 := responsePacket(req1, []byte("HTTP/1.1 100 Continue\r\n"))
+	req2 := responsePacket(resp1, []byte("a"))
+	req3 := nextPacket(req2, []byte("b"))
+	resp2 := responsePacket(req3, []byte("HTTP/1.1 200 OK\r\n\r\n"))
+
+	return []*TCPPacket{req1, req2, req3}, []*TCPPacket{resp1, resp2}
+}
+
 func TestShort100Continue(t *testing.T) {
-	listener := NewListener("", "0", EnginePcap, true, 10*time.Millisecond)
+	listener := NewListener("", "0", EnginePcap, true, 10*time.Millisecond, "", "", 0, false)
 	defer listener.Close()

-	reqPacket1 := buildPacket(true, 1, 1, []byte("POST / HTTP/1.1\r\nContent-Length: 2\r\nExpect: 100-continue\r\n\r\n"), time.Now())
-	// Packet with data have different Seq
-	reqPacket2 := buildPacket(true, 2, reqPacket1.Seq+uint32(len(reqPacket1.Data)), []byte("a"), time.Now())
-	reqPacket3 := buildPacket(true, 2, reqPacket2.Seq+1, []byte("b"), time.Now())
-
-	respPacket1 := buildPacket(false, 10, 3, []byte("HTTP/1.1 100 Continue\r\n\r\n"), time.Now())
-
-	// panic(int(uint32(len(reqPacket1.Data)) + uint32(len(reqPacket2.Data)) + uint32(len(reqPacket3.Data))))
-	respPacket2 := buildPacket(false, reqPacket3.Seq+1 /* len of data */, 2, []byte("HTTP/1.1 200 OK\r\n\r\n"), time.Now())
+	req, resp := get100ContinuePackets()

 	result := []byte("POST / HTTP/1.1\r\nContent-Length: 2\r\n\r\nab")

-	testRawListener100Continue(t, listener, result, reqPacket1, reqPacket2, reqPacket3, respPacket1, respPacket2)
+	testRawListener100Continue(t, listener, result, req[0], req[1], req[2], resp[0], resp[1])
 }

 // Response comes before Request
 func Test100ContinueWrongOrder(t *testing.T) {
-	listener := NewListener("", "0", EnginePcap, true, 10*time.Millisecond)
+	listener := NewListener("", "0", EnginePcap, true, 10*time.Millisecond, "", "", 0, false)
 	defer listener.Close()

-	reqPacket1 := buildPacket(true, 1, 1, []byte("POST / HTTP/1.1\r\nContent-Length: 2\r\nExpect: 100-continue\r\n\r\n"), time.Now())
-	// Packet with data have different Seq
-	reqPacket2 := buildPacket(true, 2, reqPacket1.Seq+uint32(len(reqPacket1.Data)), []byte("a"), time.Now())
-	reqPacket3 := buildPacket(true, 2, reqPacket2.Seq+1, []byte("b"), time.Now())
-
-	respPacket1 := buildPacket(false, 10, 3, []byte("HTTP/1.1 100 Continue\r\n"), time.Now())
-
-	// panic(int(uint32(len(reqPacket1.Data)) + uint32(len(reqPacket2.Data)) + uint32(len(reqPacket3.Data))))
-	respPacket2 := buildPacket(false, reqPacket3.Seq+1 /* len of data */, 2, []byte("HTTP/1.1 200 OK\r\n\r\n"), time.Now())
+	req, resp := get100ContinuePackets()

 	result := []byte("POST / HTTP/1.1\r\nContent-Length: 2\r\n\r\nab")

-	testRawListener100Continue(t, listener, result, respPacket1, respPacket2, reqPacket1, reqPacket2, reqPacket3)
-}
-
-func TestAlt100ContinueHeaderOrder(t *testing.T) {
-	listener := NewListener("", "0", EnginePcap, true, 10*time.Millisecond)
-	defer listener.Close()
-
-	reqPacket1 := buildPacket(true, 1, 1, []byte("POST / HTTP/1.1\r\nExpect: 100-continue\r\nContent-Length: 2\r\n\r\n"), time.Now())
-	// Packet with data have different Seq
-	reqPacket2 := buildPacket(true, 2, reqPacket1.Seq+uint32(len(reqPacket1.Data)), []byte("a"), time.Now())
-	reqPacket3 := buildPacket(true, 2, reqPacket2.Seq+1, []byte("b"), time.Now())
-
-	respPacket1 := buildPacket(false, 10, 3, []byte("HTTP/1.1 100 Continue\r\n"), time.Now())
-
-	// panic(int(uint32(len(reqPacket1.Data)) + uint32(len(reqPacket2.Data)) + uint32(len(reqPacket3.Data))))
-	respPacket2 := buildPacket(false, reqPacket3.Seq+1 /* len of data */, 2, []byte("HTTP/1.1 200 OK\r\n\r\n"), time.Now())
-
-	result := []byte("POST / HTTP/1.1\r\nContent-Length: 2\r\n\r\nab")
-
-	testRawListener100Continue(t, listener, result, reqPacket1, reqPacket2, reqPacket3, respPacket1, respPacket2)
+	testRawListener100Continue(t, listener, result, resp[0], resp[1], req[0], req[1], req[2])
 }

 func testRawListener100Continue(t *testing.T, listener *Listener, result []byte, packets ...*TCPPacket) {
@@ -349,30 +462,22 @@ func permutation(n int, list []*TCPPacket) []*TCPPacket {

 // Response comes before Request
 func TestRawListenerChunkedWrongOrder(t *testing.T) {
-	listener := NewListener("", "0", EnginePcap, true, 10*time.Millisecond)
+	listener := NewListener("", "0", EnginePcap, true, 10*time.Millisecond, "", "", 0, false)
 	defer listener.Close()

-	reqPacket1 := buildPacket(true, 1, 1, []byte("POST / HTTP/1.1\r\nTransfer-Encoding: chunked\r\nExpect: 100-continue\r\n\r\n"), time.Now())
-	// Packet with data have different Seq
-	reqPacket2 := buildPacket(true, 2, reqPacket1.Seq+uint32(len(reqPacket1.Data)), []byte("1\r\na\r\n"), time.Now())
-	reqPacket3 := buildPacket(true, 2, reqPacket2.Seq+uint32(len(reqPacket2.Data)), []byte("1\r\nb\r\n"), time.Now())
-	reqPacket4 := buildPacket(true, 2, reqPacket3.Seq+uint32(len(reqPacket3.Data)), []byte("0\r\n\r\n"), time.Now())
+	reqPacket1 := firstPacket([]byte("POST / HTTP/1.1\r\nTransfer-Encoding: chunked\r\nExpect: 100-continue\r\n\r\n"))

-	respPacket1 := buildPacket(false, 10, 3, []byte("HTTP/1.1 100 Continue\r\n\r\n"), time.Now())
+	respPacket1 := responsePacket(reqPacket1, []byte("HTTP/1.1 100 Continue\r\n"))
+	reqPacket2 := responsePacket(respPacket1, []byte("1\r\na\r\n"))
+	reqPacket3 := nextPacket(reqPacket2, []byte("1\r\nb\r\n"))
+	reqPacket4 := nextPacket(reqPacket3, []byte("0\r\n\r\n"))

-	// panic(int(uint32(len(reqPacket1.Data)) + uint32(len(reqPacket2.Data)) + uint32(len(reqPacket3.Data))))
-	respPacket2 := buildPacket(false, reqPacket4.Seq+5 /* len of data */, 2, []byte("HTTP/1.1 200 OK\r\n\r\n"), time.Now())
+	respPacket2 := responsePacket(reqPacket4, []byte("HTTP/1.1 200 OK\r\n\r\n"))

 	// Should re-construct message from all possible combinations
 	for i := 0; i < 6*5*4*3*2*1; i++ {
-
-		// if i < 54 || i > 57 {
-		// 	continue
-		// }
-
 		packets := permutation(i, []*TCPPacket{reqPacket1, reqPacket2, reqPacket3, reqPacket4, respPacket1, respPacket2})

-		t.Log("permutation:", i)
 		testChunkedSequence(t, listener, packets...)
 	}
 }
@@ -427,7 +532,7 @@ func getMessage() []*TCPPacket {

 // Response comes before Request
 func TestRawListenerBench(t *testing.T) {
-	l := NewListener("", "0", EnginePcap, true, 200*time.Millisecond)
+	l := NewListener("", "0", EnginePcap, true, 200*time.Millisecond, "", "", 0, false)
 	defer l.Close()

 	// Should re-construct message from all possible combinations
@@ -475,3 +580,44 @@ func TestRawListenerBench(t *testing.T) {
 		}
 	}
 }
+
+func TestResponseZeroContentLength(t *testing.T) {
+	var req, resp *TCPMessage
+	listener := NewListener("", "0", EnginePcap, true, 10*time.Millisecond, "", "", 0, false)
+	defer listener.Close()
+
+	reqPacket := firstPacket([]byte("POST /api/setup/install HTTP/1.1\r\nHost: localhost:22936\r\nUser-Agent: curl/7.57.0\r\nAccept: */*\r\nContent-Length: 0\r\nContent-Type: application/x-www-form-urlencoded\r\n\r\n"))
+	respPacket := responsePacket(reqPacket, []byte("HTTP/1.1 200\r\nDate: Fri, 11 May 2018 15:09:10 GMT\r\nServer: Kestrel\r\nCache-Control: no-cache\r\nTransfer-Encoding: chunked\r\n\r\n"))
+	respPacket2 := nextPacket(respPacket, []byte("0\r\n\r\n"))
+
+	// If response packet comes before request
+	listener.packetsChan <- reqPacket.dump()
+	listener.packetsChan <- respPacket.dump()
+	listener.packetsChan <- respPacket2.dump()
+
+	select {
+	case req = <-listener.messagesChan:
+	case <-time.After(time.Millisecond):
+		t.Error("Should return respose immediately")
+		return
+	}
+
+	if !req.IsIncoming {
+		t.Error("Should be request")
+	}
+
+	select {
+	case resp = <-listener.messagesChan:
+	case <-time.After(time.Millisecond):
+		t.Error("Should return response immediately")
+		return
+	}
+
+	if resp.IsIncoming {
+		t.Error("Should be response")
+	}
+
+	if !bytes.Equal(resp.UUID(), req.UUID()) {
+		t.Error("Resp and Req UUID should be equal")
+	}
+}
@@ -5,11 +5,13 @@ import (
 	"crypto/sha1"
 	"encoding/binary"
 	"encoding/hex"
-	"github.com/buger/gor/proto"
 	"log"
 	"net"
 	"strconv"
+	"strings"
 	"time"
+
+	"github.com/buger/goreplay/proto"
 )

 var _ = log.Println
@@ -47,7 +49,8 @@ type TCPMessage struct {
 	complete      bool
 }

-// NewTCPMessage pointer created from a Acknowledgment number and a channel of messages readuy to be deleted
+// NewTCPMessage pointer created from a sequence and acknowledgment numbers, whether the message is incoming and a timestamp
+// that indicates when the packet was captrued.
 func NewTCPMessage(Seq, Ack uint32, IsIncoming bool, timestamp time.Time) (msg *TCPMessage) {
 	msg = &TCPMessage{Seq: Seq, Ack: Ack, IsIncoming: IsIncoming, Start: timestamp}

@@ -72,7 +75,7 @@ func (t *TCPMessage) Bytes() (output []byte) {
 	return output
 }

-// Size returns total body size
+// BodySize returns total body size
 func (t *TCPMessage) BodySize() (size int) {
 	if len(t.packets) == 0 || t.headerPacket == -1 {
 		return 0
@@ -127,20 +130,17 @@ func (t *TCPMessage) AddPacket(packet *TCPPacket) {
 				}
 			}
 		}
-
-		if t.IsIncoming {
-			t.End = time.Now()
-		} else {
-			t.End = time.Now().Add(time.Millisecond)
-		}
-
 		if packet.OrigAck != 0 {
 			t.DataAck = packet.OrigAck
 		}

-		if packet.timestamp.Before(t.Start) {
+		if packet.timestamp.Before(t.Start) || t.Start.IsZero() {
 			t.Start = packet.timestamp
 		}
+
+		if packet.timestamp.After(t.End) || t.End.IsZero() {
+			t.End = packet.timestamp
+		}
 	}

 	t.checkSeqIntegrity()
@@ -198,7 +198,8 @@ func (t *TCPMessage) checkSeqIntegrity() {
 }

 var bEmptyLine = []byte("\r\n\r\n")
-var bChunkEnd = []byte("0\r\n\r\n")
+var bBR = []byte("\r\n")
+var bChunkEnd = []byte("\r\n0\r\n\r\n")

 func (t *TCPMessage) updateHeadersPacket() {
 	if len(t.packets) == 1 {
@@ -214,55 +215,63 @@ func (t *TCPMessage) updateHeadersPacket() {
 	}

 	for i, p := range t.packets {
-		if bytes.LastIndex(p.Data, bEmptyLine) != -1 {
-			t.headerPacket = i
-			return
+		if len(p.Data) >= len(bEmptyLine) {
+			if bytes.LastIndex(p.Data, bEmptyLine) != -1 {
+				t.headerPacket = i
+				return
+			}
+		} else if i > 0 && bytes.Equal(p.Data, bBR) {
+			idx := bytes.LastIndex(t.packets[i-1].Data, bBR)
+			if idx != -1 && idx == len(t.packets[i-1].Data)-len(bBR) {
+				t.headerPacket = i
+				return
+			}
 		}
 	}

 	return
 }

-// isMultipart returns true if message contains from multiple tcp packets
+// checkIfComplete returns true if all of the packets that compse the message arrived.
 func (t *TCPMessage) checkIfComplete() {
 	if t.seqMissing || t.headerPacket == -1 {
+		// log.Println("Seq missing", t.seqMissing, t.packets)
 		return
 	}

 	if t.methodType == httpMethodNotFound {
+		// log.Println("Method missing", t.methodType, t.packets)
 		return
 	}

 	// Responses can be emitted only if we found request
 	if !t.IsIncoming && t.AssocMessage == nil {
+		// log.Println("Assoc not found", t)
 		return
 	}

-	// If one GET, OPTIONS, or HEAD request
-	if t.methodType == httpMethodWithoutBody {
-		t.complete = true
-	} else {
-		switch t.bodyType {
-		case httpBodyEmpty:
-			t.complete = true
-		case httpBodyContentLength:
-			if t.contentLength == 0 || t.contentLength == t.BodySize() {
-				t.complete = true
-			}
-		case httpBodyChunked:
-			lastPacket := t.packets[len(t.packets)-1]
-			if bytes.LastIndex(lastPacket.Data, bChunkEnd) != -1 {
-				t.complete = true
-			}
-		default:
-			if len(t.packets) == 0 {
-				return
-			}
+	// log.Println("Found?", t)

-			last := t.packets[len(t.packets)-1]
-			if last.IsFIN {
-				t.complete = true
-			}
+	switch t.bodyType {
+	case httpBodyEmpty:
+		t.complete = true
+	case httpBodyContentLength:
+		if t.contentLength == 0 || t.contentLength == t.BodySize() {
+			t.complete = true
+		}
+	case httpBodyChunked:
+		lastPacket := t.packets[len(t.packets)-1]
+		if bytes.LastIndex(lastPacket.Data, bChunkEnd) != -1 {
+			t.complete = true
+		}
+	default:
+		if len(t.packets) == 0 {
+			return
+		}
+
+		last := t.packets[len(t.packets)-1]
+		if last.IsFIN {
+			t.complete = true
 		}
 	}
 }
@@ -270,19 +279,11 @@ func (t *TCPMessage) checkIfComplete() {
 type httpMethodType uint8

 const (
-	httpMethodNotSet      httpMethodType = 0
-	httpMethodWithBody    httpMethodType = 1
-	httpMethodWithoutBody httpMethodType = 2
-	httpMethodNotFound    httpMethodType = 3
+	httpMethodNotSet   httpMethodType = 0
+	httpMethodKnown    httpMethodType = 1
+	httpMethodNotFound httpMethodType = 2
 )

-var methodsWithBody = [][]byte{
-	[]byte("POST"),
-	[]byte("PUT"),
-	[]byte("PATCH"),
-	[]byte("CONNECT"),
-}
-
 func (t *TCPMessage) updateMethodType() {
 	// if there is cache
 	if t.methodType != httpMethodNotSet && t.methodType != httpMethodNotFound {
@@ -299,11 +300,7 @@ func (t *TCPMessage) updateMethodType() {
 	}

 	if t.IsIncoming {
-		var method []byte
-
 		if mIdx := bytes.IndexByte(d[:8], ' '); mIdx != -1 {
-			method = d[:mIdx]
-
 			// Check that after method we have absolute or relative path
 			switch d[mIdx+1] {
 			case '/', 'h', '*':
@@ -316,21 +313,14 @@ func (t *TCPMessage) updateMethodType() {
 			return
 		}

-		for _, m := range methodsWithBody {
-			if len(m) == len(method) && bytes.Equal(m, method) {
-				t.methodType = httpMethodWithBody
-				return
-			}
-		}
-
-		t.methodType = httpMethodWithoutBody
+		t.methodType = httpMethodKnown
 	} else {
 		if !bytes.Equal(d[:6], []byte("HTTP/1")) {
 			t.methodType = httpMethodNotFound
 			return
 		}

-		t.methodType = httpMethodWithBody
+		t.methodType = httpMethodKnown
 	}
 }

@@ -357,7 +347,7 @@ func (t *TCPMessage) updateBodyType() {

 	var lengthB, encB, connB []byte

-	proto.ParseHeaders(t.packetsData(), func(header, value []byte)bool{
+	proto.ParseHeaders(t.packetsData(), func(header, value []byte) bool {
 		if proto.HeadersEqual(header, []byte("Content-Length")) {
 			lengthB = value
 			return false
@@ -378,13 +368,20 @@ func (t *TCPMessage) updateBodyType() {
 	switch t.methodType {
 	case httpMethodNotFound:
 		return
-	case httpMethodWithoutBody:
-		t.bodyType = httpBodyEmpty
-		return
-	case httpMethodWithBody:
+	case httpMethodKnown:
+
+		if !t.IsIncoming &&
+			t.AssocMessage != nil &&
+			bytes.IndexByte(t.AssocMessage.Bytes(), ' ') > -1 &&
+			bytes.Equal([]byte("HEAD"), proto.Method(t.AssocMessage.Bytes())) {
+			// Need to check if this is a response to a head request,
+			// in which case the body has to be empty regardless.
+			t.bodyType = httpBodyEmpty
+			return
+		}
+
 		if len(lengthB) > 0 {
 			t.contentLength, _ = strconv.Atoi(string(lengthB))
-
 			if t.contentLength == 0 {
 				t.bodyType = httpBodyEmpty
 			} else {
@@ -423,10 +420,6 @@ func (t *TCPMessage) check100Continue() {
 		return
 	}

-	if t.methodType != httpMethodWithBody {
-		return
-	}
-
 	if t.seqMissing || t.headerPacket == -1 {
 		return
 	}
@@ -438,7 +431,7 @@ func (t *TCPMessage) check100Continue() {
 	}

 	var expectB []byte
-	proto.ParseHeaders(t.packetsData(), func(header, value []byte)bool{
+	proto.ParseHeaders(t.packetsData(), func(header, value []byte) bool {
 		if proto.HeadersEqual(header, bExpectHeader) {
 			expectB = value
 			return false
@@ -509,3 +502,11 @@ func (t *TCPMessage) ID() tcpID {
 func (t *TCPMessage) IP() net.IP {
 	return net.IP(t.packets[0].Addr)
 }
+
+func (t *TCPMessage) String() string {
+	return strings.Join([]string{
+		"Len packets: " + strconv.Itoa(len(t.packets)),
+		"Data size:" + strconv.Itoa(len(t.Bytes())),
+		"Data:" + string(t.Bytes()),
+	}, "\n")
+}
@@ -170,16 +170,16 @@ func TestTCPMessageMethodType(t *testing.T) {
 		payload            string
 		expectedMethodType httpMethodType
 	}{
-		{true, "GET / HTTP/1.1\r\n\r\n", httpMethodWithoutBody},
-		{true, "GET * HTTP/1.1\r\n\r\n", httpMethodWithoutBody},
-		{true, "UNKNOWN / HTTP/1.1\r\n\r\n", httpMethodWithoutBody},
-		{true, "GET http://example.com HTTP/1.1\r\n\r\n", httpMethodWithoutBody},
-		{true, "POST / HTTP/1.1\r\n\r\n", httpMethodWithBody},
-		{true, "PUT / HTTP/1.1\r\n\r\n", httpMethodWithBody},
+		{true, "GET / HTTP/1.1\r\n\r\n", httpMethodKnown},
+		{true, "GET * HTTP/1.1\r\n\r\n", httpMethodKnown},
+		{true, "UNKNOWN / HTTP/1.1\r\n\r\n", httpMethodKnown},
+		{true, "GET http://example.com HTTP/1.1\r\n\r\n", httpMethodKnown},
+		{true, "POST / HTTP/1.1\r\n\r\n", httpMethodKnown},
+		{true, "PUT / HTTP/1.1\r\n\r\n", httpMethodKnown},
 		{true, "GET zxc HTTP/1.1\r\n\r\n", httpMethodNotFound},
 		{true, "GET / HTTP\r\n\r\n", httpMethodNotFound},
 		{true, "VERYLONGMETHOD / HTTP/1.1\r\n\r\n", httpMethodNotFound},
-		{false, "HTTP/1.1 200 OK\r\n\r\n", httpMethodWithBody},
+		{false, "HTTP/1.1 200 OK\r\n\r\n", httpMethodKnown},
 		{false, "HTTP /1.1 200 OK\r\n\r\n", httpMethodNotFound},
 	}

@@ -199,6 +199,7 @@ func TestTCPMessageBodyType(t *testing.T) {
 		expectedBodyType httpBodyType
 	}{
 		{true, "GET / HTTP/1.1\r\n\r\n", httpBodyEmpty},
+		{true, "GET / HTTP/1.1\r\nContent-Length: 2\r\n\r\nab", httpBodyContentLength},
 		{true, "POST / HTTP/1.1\r\n\r\n", httpBodyEmpty},
 		{true, "POST / HTTP/1.1\r\nUser-Agent: zxc\r\n\r\n", httpBodyEmpty},
 		{false, "HTTP/1.1 200 OK\r\n\r\n", httpBodyEmpty},
@@ -255,4 +256,3 @@ func TestTcpMessageStart(t *testing.T) {
 		t.Error("Message timestamp should be equal to the lowest related packet timestamp", start, msg.Start)
 	}
 }
-
@@ -36,11 +36,11 @@ type TCPPacket struct {
 	DataOffset uint8
 	IsFIN      bool

-	Raw  []byte
-	Data []byte
-	Addr []byte
+	Raw       []byte
+	Data      []byte
+	Addr      []byte
 	timestamp time.Time
-	ID   tcpID
+	ID        tcpID
 }

 // ParseTCPPacket takes address and tcp payload and returns parsed TCPPacket
@@ -77,15 +77,15 @@ func (t *TCPPacket) ParseBasic() {
 	t.DataOffset = (t.Raw[12] & 0xF0) >> 4
 	t.IsFIN = t.Raw[13]&0x01 != 0

-	// log.Println("DataOffset:", t.DataOffset, t.DestPort, t.SrcPort, t.Seq, t.Ack)
-
-	t.Data = t.Raw[t.DataOffset*4:]
+	if len(t.Raw) >= int(t.DataOffset*4) {
+		t.Data = t.Raw[t.DataOffset*4:]
+	}
 }

 func (t *TCPPacket) dump() *packet {

 	packetSrcIP := make([]byte, 16)
-	packetData := make([]byte, len(t.Data) + 16)
+	packetData := make([]byte, len(t.Data)+16)

 	copy(packetSrcIP, t.Addr)

@@ -104,9 +104,9 @@ func (t *TCPPacket) dump() *packet {
 	copy(packetData[16:], t.Data)

 	return &packet{
-		srcIP: packetSrcIP,
-		data:packetData,
-		timestamp:t.timestamp,
+		srcIP:     packetSrcIP,
+		data:      packetData,
+		timestamp: t.timestamp,
 	}

 }
@@ -30,6 +30,8 @@ type AppSettings struct {
 	stats     bool
 	exitAfter time.Duration

+	pprof string
+
 	splitOutput bool

 	inputDummy   MultiOption
@@ -37,9 +39,11 @@ type AppSettings struct {
 	outputStdout bool
 	outputNull   bool

-	inputTCP       MultiOption
-	outputTCP      MultiOption
-	outputTCPStats bool
+	inputTCP        MultiOption
+	inputTCPConfig  TCPInputConfig
+	outputTCP       MultiOption
+	outputTCPConfig TCPOutputConfig
+	outputTCPStats  bool

 	inputFile        MultiOption
 	inputFileLoop    bool
@@ -50,12 +54,21 @@ type AppSettings struct {
 	inputRAWEngine        string
 	inputRAWTrackResponse bool
 	inputRAWRealIPHeader  string
+	inputRAWExpire        time.Duration
+	inputRAWBpfFilter     string
+	inputRAWTimestampType string
+	copyBufferSize        int
+	inputRAWImmediateMode bool
+	inputRawBufferSize    int
+	inputRAWOverrideSnapLen bool

 	middleware string

 	inputHTTP  MultiOption
 	outputHTTP MultiOption

+	prettifyHTTP bool
+
 	outputHTTPConfig HTTPOutputConfig
 	modifierConfig   HTTPModifierConfig

@@ -75,6 +88,7 @@ func usage() {
 func init() {
 	flag.Usage = usage

+	flag.StringVar(&Settings.pprof, "http-pprof", "", "Enable profiling. Starts  http server on specified port, exposing special /debug/pprof endpoint. Example: `:8181`")
 	flag.BoolVar(&Settings.verbose, "verbose", false, "Turn on more verbose output")
 	flag.BoolVar(&Settings.debug, "debug", false, "Turn on debug output, shows all intercepted traffic. Works only when with `verbose` flag")
 	flag.BoolVar(&Settings.stats, "stats", false, "Turn on queue stats output")
@@ -90,7 +104,12 @@ func init() {
 	flag.BoolVar(&Settings.outputNull, "output-null", false, "Used for testing inputs. Drops all requests.")

 	flag.Var(&Settings.inputTCP, "input-tcp", "Used for internal communication between Gor instances. Example: \n\t# Receive requests from other Gor instances on 28020 port, and redirect output to staging\n\tgor --input-tcp :28020 --output-http staging.com")
+	flag.BoolVar(&Settings.inputTCPConfig.secure, "input-tcp-secure", false, "Turn on TLS security. Do not forget to specify certificate and key files.")
+	flag.StringVar(&Settings.inputTCPConfig.certificatePath, "input-tcp-certificate", "", "Path to PEM encoded certificate file. Used when TLS turned on.")
+	flag.StringVar(&Settings.inputTCPConfig.keyPath, "input-tcp-certificate-key", "", "Path to PEM encoded certificate key file. Used when TLS turned on.")
+
 	flag.Var(&Settings.outputTCP, "output-tcp", "Used for internal communication between Gor instances. Example: \n\t# Listen for requests on 80 port and forward them to other Gor instance on 28020 port\n\tgor --input-raw :80 --output-tcp replay.local:28020")
+	flag.BoolVar(&Settings.outputTCPConfig.secure, "output-tcp-secure", false, "Use TLS secure connection. --input-file on another end should have TLS turned on as well.")
 	flag.BoolVar(&Settings.outputTCPStats, "output-tcp-stats", false, "Report TCP output queue stats to console every 5 seconds.")

 	flag.Var(&Settings.inputFile, "input-file", "Read requests from file: \n\tgor --input-file ./requests.gor --output-http staging.com")
@@ -104,6 +123,10 @@ func init() {
 	Settings.outputFileConfig.sizeLimit.Set("32mb")
 	flag.Var(&Settings.outputFileConfig.sizeLimit, "output-file-size-limit", "Size of each chunk. Default: 32mb")
 	flag.IntVar(&Settings.outputFileConfig.queueLimit, "output-file-queue-limit", 256, "The length of the chunk queue. Default: 256")
+	Settings.outputFileConfig.outputFileMaxSize.Set("-1")
+	flag.Var(&Settings.outputFileConfig.outputFileMaxSize, "output-file-max-size-limit", "Max size of output file, Default: 1TB")
+
+	flag.BoolVar(&Settings.prettifyHTTP, "prettify-http", false, "If enabled, will automatically decode requests and responses with: Content-Encodning: gzip and Transfer-Encoding: chunked. Useful for debugging, in conjuction with --output-stdout")

 	flag.Var(&Settings.inputRAW, "input-raw", "Capture traffic from given port (use RAW sockets and require *sudo* access):\n\t# Capture traffic from 8080 port\n\tgor --input-raw :8080 --output-http staging.com")

@@ -113,17 +136,34 @@ func init() {

 	flag.StringVar(&Settings.inputRAWRealIPHeader, "input-raw-realip-header", "", "If not blank, injects header with given name and real IP value to the request payload. Usually this header should be named: X-Real-IP")

+	flag.DurationVar(&Settings.inputRAWExpire, "input-raw-expire", time.Second*2, "How much it should wait for the last TCP packet, till consider that TCP message complete.")
+
+	flag.StringVar(&Settings.inputRAWBpfFilter, "input-raw-bpf-filter", "", "BPF filter to write custom expressions. Can be useful in case of non standard network interfaces like tunneling or SPAN port. Example: --input-raw-bpf-filter 'dst port 80'")
+
+	flag.StringVar(&Settings.inputRAWTimestampType, "input-raw-timestamp-type", "", "Possible values: PCAP_TSTAMP_HOST, PCAP_TSTAMP_HOST_LOWPREC, PCAP_TSTAMP_HOST_HIPREC, PCAP_TSTAMP_ADAPTER, PCAP_TSTAMP_ADAPTER_UNSYNCED. This values not supported on all systems, GoReplay will tell you available values of you put wrong one.")
+	flag.IntVar(&Settings.copyBufferSize, "copy-buffer-size", 5*1024*1024, "Set the buffer size for an individual request (default 5M)")
+	flag.BoolVar(&Settings.inputRAWOverrideSnapLen, "input-raw-override-snaplen", false, "Override the capture snaplen to be 64k. Required for some Virtualized environments")
+	flag.BoolVar(&Settings.inputRAWImmediateMode, "input-raw-immediate-mode", false, "Set pcap interface to immediate mode.")
+
+	flag.IntVar(&Settings.inputRawBufferSize, "input-raw-buffer-size", 0, "Controls size of the OS buffer (in bytes) which holds packets until they dispatched. Default value depends by system: in Linux around 2MB. If you see big package drop, increase this value.")
+
 	flag.StringVar(&Settings.middleware, "middleware", "", "Used for modifying traffic using external command")

 	// flag.Var(&Settings.inputHTTP, "input-http", "Read requests from HTTP, should be explicitly sent from your application:\n\t# Listen for http on 9000\n\tgor --input-http :9000 --output-http staging.com")

 	flag.Var(&Settings.outputHTTP, "output-http", "Forwards incoming requests to given http address.\n\t# Redirect all incoming requests to staging.com address \n\tgor --input-raw :80 --output-http http://staging.com")
 	flag.IntVar(&Settings.outputHTTPConfig.BufferSize, "output-http-response-buffer", 0, "HTTP response buffer size, all data after this size will be discarded.")
-	flag.IntVar(&Settings.outputHTTPConfig.workers, "output-http-workers", 0, "Gor uses dynamic worker scaling by default.  Enter a number to run a set number of workers.")
+
+  flag.IntVar(&Settings.outputHTTPConfig.workersMin, "output-http-workers-min", 0, "Gor uses dynamic worker scaling. Enter a number to set a minimum number of workers. default = 1.")
+	flag.IntVar(&Settings.outputHTTPConfig.workersMax, "output-http-workers", 0, "Gor uses dynamic worker scaling. Enter a number to set a maximum number of workers. default = 0 = unlimited.")
+	flag.IntVar(&Settings.outputHTTPConfig.queueLen, "output-http-queue-len", 1000, "Number of requests that can be queued for output, if all workers are busy. default = 1000")
+
 	flag.IntVar(&Settings.outputHTTPConfig.redirectLimit, "output-http-redirects", 0, "Enable how often redirects should be followed.")
 	flag.DurationVar(&Settings.outputHTTPConfig.Timeout, "output-http-timeout", 5*time.Second, "Specify HTTP request/response timeout. By default 5s. Example: --output-http-timeout 30s")
+	flag.BoolVar(&Settings.outputHTTPConfig.TrackResponses, "output-http-track-response", false, "If turned on, HTTP output responses will be set to all outputs like stdout, file and etc.")

-	flag.BoolVar(&Settings.outputHTTPConfig.stats, "output-http-stats", false, "Report http output queue stats to console every 5 seconds.")
+	flag.BoolVar(&Settings.outputHTTPConfig.stats, "output-http-stats", false, "Report http output queue stats to console every N milliseconds. See output-http-stats-ms")
+	flag.IntVar(&Settings.outputHTTPConfig.statsMs, "output-http-stats-ms", 5000, "Report http output queue stats to console every N milliseconds. default: 5000")
 	flag.BoolVar(&Settings.outputHTTPConfig.OriginalHost, "http-original-host", false, "Normally gor replaces the Host http header with the host supplied with --output-http.  This option disables that behavior, preserving the original Host header.")
 	flag.BoolVar(&Settings.outputHTTPConfig.Debug, "output-http-debug", false, "Enables http debug output.")

@@ -140,6 +180,8 @@ func init() {
 	flag.Var(&Settings.modifierConfig.headers, "http-set-header", "Inject additional headers to http reqest:\n\tgor --input-raw :8080 --output-http staging.com --http-set-header 'User-Agent: Gor'")
 	flag.Var(&Settings.modifierConfig.headers, "output-http-header", "WARNING: `--output-http-header` DEPRECATED, use `--http-set-header` instead")

+	flag.Var(&Settings.modifierConfig.headerRewrite, "http-rewrite-header", "Rewrite the request header based on a mapping:\n\tgor --input-raw :8080 --output-http staging.com --http-rewrite-header Host: (.*).example.com,$1.beta.example.com")
+
 	flag.Var(&Settings.modifierConfig.params, "http-set-param", "Set request url param, if param already exists it will be overwritten:\n\tgor --input-raw :8080 --output-http staging.com --http-set-param api_key=1")

 	flag.Var(&Settings.modifierConfig.methods, "http-allow-method", "Whitelist of HTTP methods to replay. Anything else will be dropped:\n\tgor --input-raw :8080 --output-http staging.com --http-allow-method GET --http-allow-method OPTIONS")
@@ -158,7 +200,10 @@ func init() {

 	flag.Var(&Settings.modifierConfig.headerNegativeFilters, "http-disallow-header", "A regexp to match a specific header against. Requests with matching headers will be dropped:\n\t gor --input-raw :8080 --output-http staging.com --http-disallow-header \"User-Agent: Replayed by Gor\"")

-	flag.Var(&Settings.modifierConfig.headerHashFilters, "http-header-limiter", "Takes a fraction of requests, consistently taking or rejecting a request based on the FNV32-1A hash of a specific header:\n\t gor --input-raw :8080 --output-http staging.com --http-header-imiter user-id:25%")
+	flag.Var(&Settings.modifierConfig.headerBasicAuthFilters, "http-basic-auth-filter", "A regexp to match the decoded basic auth string against. Requests with non-matching headers will be dropped:\n\t gor --input-raw :8080 --output-http staging.com --http-basic-auth-filter \"^customer[0-9].*\"")
+
+	flag.Var(&Settings.modifierConfig.headerHashFilters, "http-header-limiter", "Takes a fraction of requests, consistently taking or rejecting a request based on the FNV32-1A hash of a specific header:\n\t gor --input-raw :8080 --output-http staging.com --http-header-limiter user-id:25%")
+
 	flag.Var(&Settings.modifierConfig.headerHashFilters, "output-http-header-hash-filter", "WARNING: `output-http-header-hash-filter` DEPRECATED, use `--http-header-hash-limiter` instead")

 	flag.Var(&Settings.modifierConfig.paramHashFilters, "http-param-limiter", "Takes a fraction of requests, consistently taking or rejecting a request based on the FNV32-1A hash of a specific GET param:\n\t gor --input-raw :8080 --output-http staging.com --http-param-limiter user_id:25%")
@@ -0,0 +1,39 @@
+name: goreplay
+version: '1.0'
+summary: GoReplay is an open-source tool for capturing and replaying live HTTP traffic 
+description: |
+  GoReplay is an open-source tool for capturing and replaying 
+  live HTTP traffic into a test environment in order to continuously 
+  test your system with real data. It can be used to increase confidence 
+  in code deployments, configuration changes and infrastructure changes. 
+grade: stable
+confinement: strict
+base: core18
+parts:
+  goreplay:
+    plugin: go
+    source: https://github.com/buger/goreplay.git
+    go-importpath: github.com/buger/goreplay
+    build-packages:
+      - build-essential
+      - libpcap-dev
+    stage-packages:
+      - libpcap0.8
+
+apps:
+  goreplay:
+    command: bin/goreplay
+    daemon: simple
+    restart-condition: on-abnormal
+    plugs:
+      - home
+      - network
+      - network-bind
+      - network-control
+      - network-observe
+      - netlink-connector
+      - netlink-audit
+      - bluetooth-control
+      - firewall-control
+      - x11
+      
@@ -8,7 +8,7 @@ import (

 // TestInput used for testing purpose, it allows emitting requests on demand
 type TestInput struct {
-	data chan []byte
+	data       chan []byte
 	skipHeader bool
 }

@@ -0,0 +1,350 @@
+# Changelog
+
+#### Version 1.11.0 (2016-12-20)
+
+New Features:
+ - Metrics! Thanks to Sébastien Launay for all his work on this feature
+   ([#701](https://github.com/Shopify/sarama/pull/701),
+    [#746](https://github.com/Shopify/sarama/pull/746),
+    [#766](https://github.com/Shopify/sarama/pull/766)).
+ - Add support for LZ4 compression
+   ([#786](https://github.com/Shopify/sarama/pull/786)).
+ - Add support for ListOffsetRequest v1 and Kafka 0.10.1
+   ([#775](https://github.com/Shopify/sarama/pull/775)).
+ - Added a `HighWaterMarks` method to the Consumer which aggregates the
+   `HighWaterMarkOffset` values of its child topic/partitions
+   ([#769](https://github.com/Shopify/sarama/pull/769)).
+
+Bug Fixes:
+ - Fixed producing when using timestamps, compression and Kafka 0.10
+   ([#759](https://github.com/Shopify/sarama/pull/759)).
+ - Added missing decoder methods to DescribeGroups response
+   ([#756](https://github.com/Shopify/sarama/pull/756)).
+ - Fix producer shutdown when `Return.Errors` is disabled
+   ([#787](https://github.com/Shopify/sarama/pull/787)).
+ - Don't mutate configuration in SyncProducer
+   ([#790](https://github.com/Shopify/sarama/pull/790)).
+ - Fix crash on SASL initialization failure
+   ([#795](https://github.com/Shopify/sarama/pull/795)).
+
+#### Version 1.10.1 (2016-08-30)
+
+Bug Fixes:
+ - Fix the documentation for `HashPartitioner` which was incorrect
+   ([#717](https://github.com/Shopify/sarama/pull/717)).
+ - Permit client creation even when it is limited by ACLs
+   ([#722](https://github.com/Shopify/sarama/pull/722)).
+ - Several fixes to the consumer timer optimization code, regressions introduced
+   in v1.10.0. Go's timers are finicky
+   ([#730](https://github.com/Shopify/sarama/pull/730),
+    [#733](https://github.com/Shopify/sarama/pull/733),
+    [#734](https://github.com/Shopify/sarama/pull/734)).
+ - Handle consuming compressed relative offsets with Kafka 0.10
+   ([#735](https://github.com/Shopify/sarama/pull/735)).
+
+#### Version 1.10.0 (2016-08-02)
+
+_Important:_ As of Sarama 1.10 it is necessary to tell Sarama the version of
+Kafka you are running against (via the `config.Version` value) in order to use
+features that may not be compatible with old Kafka versions. If you don't
+specify this value it will default to 0.8.2 (the minimum supported), and trying
+to use more recent features (like the offset manager) will fail with an error.
+
+_Also:_ The offset-manager's behaviour has been changed to match the upstream
+java consumer (see [#705](https://github.com/Shopify/sarama/pull/705) and
+[#713](https://github.com/Shopify/sarama/pull/713)). If you use the
+offset-manager, please ensure that you are committing one *greater* than the
+last consumed message offset or else you may end up consuming duplicate
+messages.
+
+New Features:
+ - Support for Kafka 0.10
+   ([#672](https://github.com/Shopify/sarama/pull/672),
+    [#678](https://github.com/Shopify/sarama/pull/678),
+    [#681](https://github.com/Shopify/sarama/pull/681), and others).
+ - Support for configuring the target Kafka version
+   ([#676](https://github.com/Shopify/sarama/pull/676)).
+ - Batch producing support in the SyncProducer
+   ([#677](https://github.com/Shopify/sarama/pull/677)).
+ - Extend producer mock to allow setting expectations on message contents
+   ([#667](https://github.com/Shopify/sarama/pull/667)).
+
+Improvements:
+ - Support `nil` compressed messages for deleting in compacted topics
+   ([#634](https://github.com/Shopify/sarama/pull/634)).
+ - Pre-allocate decoding errors, greatly reducing heap usage and GC time against
+   misbehaving brokers ([#690](https://github.com/Shopify/sarama/pull/690)).
+ - Re-use consumer expiry timers, removing one allocation per consumed message
+   ([#707](https://github.com/Shopify/sarama/pull/707)).
+
+Bug Fixes:
+ - Actually default the client ID to "sarama" like we say we do
+   ([#664](https://github.com/Shopify/sarama/pull/664)).
+ - Fix a rare issue where `Client.Leader` could return the wrong error
+   ([#685](https://github.com/Shopify/sarama/pull/685)).
+ - Fix a possible tight loop in the consumer
+   ([#693](https://github.com/Shopify/sarama/pull/693)).
+ - Match upstream's offset-tracking behaviour
+   ([#705](https://github.com/Shopify/sarama/pull/705)).
+ - Report UnknownTopicOrPartition errors from the offset manager
+   ([#706](https://github.com/Shopify/sarama/pull/706)).
+ - Fix possible negative partition value from the HashPartitioner
+   ([#709](https://github.com/Shopify/sarama/pull/709)).
+
+#### Version 1.9.0 (2016-05-16)
+
+New Features:
+ - Add support for custom offset manager retention durations
+   ([#602](https://github.com/Shopify/sarama/pull/602)).
+ - Publish low-level mocks to enable testing of third-party producer/consumer
+   implementations ([#570](https://github.com/Shopify/sarama/pull/570)).
+ - Declare support for Golang 1.6
+   ([#611](https://github.com/Shopify/sarama/pull/611)).
+ - Support for SASL plain-text auth
+   ([#648](https://github.com/Shopify/sarama/pull/648)).
+
+Improvements:
+ - Simplified broker locking scheme slightly
+   ([#604](https://github.com/Shopify/sarama/pull/604)).
+ - Documentation cleanup
+   ([#605](https://github.com/Shopify/sarama/pull/605),
+    [#621](https://github.com/Shopify/sarama/pull/621),
+    [#654](https://github.com/Shopify/sarama/pull/654)).
+
+Bug Fixes:
+ - Fix race condition shutting down the OffsetManager
+   ([#658](https://github.com/Shopify/sarama/pull/658)).
+
+#### Version 1.8.0 (2016-02-01)
+
+New Features:
+ - Full support for Kafka 0.9:
+   - All protocol messages and fields
+   ([#586](https://github.com/Shopify/sarama/pull/586),
+   [#588](https://github.com/Shopify/sarama/pull/588),
+   [#590](https://github.com/Shopify/sarama/pull/590)).
+   - Verified that TLS support works
+   ([#581](https://github.com/Shopify/sarama/pull/581)).
+   - Fixed the OffsetManager compatibility
+   ([#585](https://github.com/Shopify/sarama/pull/585)).
+
+Improvements:
+ - Optimize for fewer system calls when reading from the network
+   ([#584](https://github.com/Shopify/sarama/pull/584)).
+ - Automatically retry `InvalidMessage` errors to match upstream behaviour
+   ([#589](https://github.com/Shopify/sarama/pull/589)).
+
+#### Version 1.7.0 (2015-12-11)
+
+New Features:
+ - Preliminary support for Kafka 0.9
+   ([#572](https://github.com/Shopify/sarama/pull/572)). This comes with several
+   caveats:
+   - Protocol-layer support is mostly in place
+     ([#577](https://github.com/Shopify/sarama/pull/577)), however Kafka 0.9
+     renamed some messages and fields, which we did not in order to preserve API
+     compatibility.
+   - The producer and consumer work against 0.9, but the offset manager does
+     not ([#573](https://github.com/Shopify/sarama/pull/573)).
+   - TLS support may or may not work
+     ([#581](https://github.com/Shopify/sarama/pull/581)).
+
+Improvements:
+ - Don't wait for request timeouts on dead brokers, greatly speeding recovery
+   when the TCP connection is left hanging
+   ([#548](https://github.com/Shopify/sarama/pull/548)).
+ - Refactored part of the producer. The new version provides a much more elegant
+   solution to [#449](https://github.com/Shopify/sarama/pull/449). It is also
+   slightly more efficient, and much more precise in calculating batch sizes
+   when compression is used
+   ([#549](https://github.com/Shopify/sarama/pull/549),
+   [#550](https://github.com/Shopify/sarama/pull/550),
+   [#551](https://github.com/Shopify/sarama/pull/551)).
+
+Bug Fixes:
+ - Fix race condition in consumer test mock
+   ([#553](https://github.com/Shopify/sarama/pull/553)).
+
+#### Version 1.6.1 (2015-09-25)
+
+Bug Fixes:
+ - Fix panic that could occur if a user-supplied message value failed to encode
+   ([#449](https://github.com/Shopify/sarama/pull/449)).
+
+#### Version 1.6.0 (2015-09-04)
+
+New Features:
+ - Implementation of a consumer offset manager using the APIs introduced in
+   Kafka 0.8.2. The API is designed mainly for integration into a future
+   high-level consumer, not for direct use, although it is *possible* to use it
+   directly.
+   ([#461](https://github.com/Shopify/sarama/pull/461)).
+
+Improvements:
+ - CRC32 calculation is much faster on machines with SSE4.2 instructions,
+   removing a major hotspot from most profiles
+   ([#255](https://github.com/Shopify/sarama/pull/255)).
+
+Bug Fixes:
+ - Make protocol decoding more robust against some malformed packets generated
+   by go-fuzz ([#523](https://github.com/Shopify/sarama/pull/523),
+   [#525](https://github.com/Shopify/sarama/pull/525)) or found in other ways
+   ([#528](https://github.com/Shopify/sarama/pull/528)).
+ - Fix a potential race condition panic in the consumer on shutdown
+   ([#529](https://github.com/Shopify/sarama/pull/529)).
+
+#### Version 1.5.0 (2015-08-17)
+
+New Features:
+ - TLS-encrypted network connections are now supported. This feature is subject
+   to change when Kafka releases built-in TLS support, but for now this is
+   enough to work with TLS-terminating proxies
+   ([#154](https://github.com/Shopify/sarama/pull/154)).
+
+Improvements:
+ - The consumer will not block if a single partition is not drained by the user;
+   all other partitions will continue to consume normally
+   ([#485](https://github.com/Shopify/sarama/pull/485)).
+ - Formatting of error strings has been much improved
+   ([#495](https://github.com/Shopify/sarama/pull/495)).
+ - Internal refactoring of the producer for code cleanliness and to enable
+   future work ([#300](https://github.com/Shopify/sarama/pull/300)).
+
+Bug Fixes:
+ - Fix a potential deadlock in the consumer on shutdown
+   ([#475](https://github.com/Shopify/sarama/pull/475)).
+
+#### Version 1.4.3 (2015-07-21)
+
+Bug Fixes:
+ - Don't include the partitioner in the producer's "fetch partitions"
+   circuit-breaker ([#466](https://github.com/Shopify/sarama/pull/466)).
+ - Don't retry messages until the broker is closed when abandoning a broker in
+   the producer ([#468](https://github.com/Shopify/sarama/pull/468)).
+ - Update the import path for snappy-go, it has moved again and the API has
+   changed slightly ([#486](https://github.com/Shopify/sarama/pull/486)).
+
+#### Version 1.4.2 (2015-05-27)
+
+Bug Fixes:
+ - Update the import path for snappy-go, it has moved from google code to github
+   ([#456](https://github.com/Shopify/sarama/pull/456)).
+
+#### Version 1.4.1 (2015-05-25)
+
+Improvements:
+ - Optimizations when decoding snappy messages, thanks to John Potocny
+   ([#446](https://github.com/Shopify/sarama/pull/446)).
+
+Bug Fixes:
+ - Fix hypothetical race conditions on producer shutdown
+   ([#450](https://github.com/Shopify/sarama/pull/450),
+   [#451](https://github.com/Shopify/sarama/pull/451)).
+
+#### Version 1.4.0 (2015-05-01)
+
+New Features:
+ - The consumer now implements `Topics()` and `Partitions()` methods to enable
+   users to dynamically choose what topics/partitions to consume without
+   instantiating a full client
+   ([#431](https://github.com/Shopify/sarama/pull/431)).
+ - The partition-consumer now exposes the high water mark offset value returned
+   by the broker via the `HighWaterMarkOffset()` method ([#339](https://github.com/Shopify/sarama/pull/339)).
+ - Added a `kafka-console-consumer` tool capable of handling multiple
+   partitions, and deprecated the now-obsolete `kafka-console-partitionConsumer`
+   ([#439](https://github.com/Shopify/sarama/pull/439),
+   [#442](https://github.com/Shopify/sarama/pull/442)).
+
+Improvements:
+ - The producer's logging during retry scenarios is more consistent, more
+   useful, and slightly less verbose
+   ([#429](https://github.com/Shopify/sarama/pull/429)).
+ - The client now shuffles its initial list of seed brokers in order to prevent
+   thundering herd on the first broker in the list
+   ([#441](https://github.com/Shopify/sarama/pull/441)).
+
+Bug Fixes:
+ - The producer now correctly manages its state if retries occur when it is
+   shutting down, fixing several instances of confusing behaviour and at least
+   one potential deadlock ([#419](https://github.com/Shopify/sarama/pull/419)).
+ - The consumer now handles messages for different partitions asynchronously,
+   making it much more resilient to specific user code ordering
+   ([#325](https://github.com/Shopify/sarama/pull/325)).
+
+#### Version 1.3.0 (2015-04-16)
+
+New Features:
+ - The client now tracks consumer group coordinators using
+   ConsumerMetadataRequests similar to how it tracks partition leadership using
+   regular MetadataRequests ([#411](https://github.com/Shopify/sarama/pull/411)).
+   This adds two methods to the client API:
+   - `Coordinator(consumerGroup string) (*Broker, error)`
+   - `RefreshCoordinator(consumerGroup string) error`
+
+Improvements:
+ - ConsumerMetadataResponses now automatically create a Broker object out of the
+   ID/address/port combination for the Coordinator; accessing the fields
+   individually has been deprecated
+   ([#413](https://github.com/Shopify/sarama/pull/413)).
+ - Much improved handling of `OffsetOutOfRange` errors in the consumer.
+   Consumers will fail to start if the provided offset is out of range
+   ([#418](https://github.com/Shopify/sarama/pull/418))
+   and they will automatically shut down if the offset falls out of range
+   ([#424](https://github.com/Shopify/sarama/pull/424)).
+ - Small performance improvement in encoding and decoding protocol messages
+   ([#427](https://github.com/Shopify/sarama/pull/427)).
+
+Bug Fixes:
+ - Fix a rare race condition in the client's background metadata refresher if
+   it happens to be activated while the client is being closed
+   ([#422](https://github.com/Shopify/sarama/pull/422)).
+
+#### Version 1.2.0 (2015-04-07)
+
+Improvements:
+ - The producer's behaviour when `Flush.Frequency` is set is now more intuitive
+   ([#389](https://github.com/Shopify/sarama/pull/389)).
+ - The producer is now somewhat more memory-efficient during and after retrying
+   messages due to an improved queue implementation
+   ([#396](https://github.com/Shopify/sarama/pull/396)).
+ - The consumer produces much more useful logging output when leadership
+   changes ([#385](https://github.com/Shopify/sarama/pull/385)).
+ - The client's `GetOffset` method will now automatically refresh metadata and
+   retry once in the event of stale information or similar
+   ([#394](https://github.com/Shopify/sarama/pull/394)).
+ - Broker connections now have support for using TCP keepalives
+   ([#407](https://github.com/Shopify/sarama/issues/407)).
+
+Bug Fixes:
+ - The OffsetCommitRequest message now correctly implements all three possible
+   API versions ([#390](https://github.com/Shopify/sarama/pull/390),
+   [#400](https://github.com/Shopify/sarama/pull/400)).
+
+#### Version 1.1.0 (2015-03-20)
+
+Improvements:
+ - Wrap the producer's partitioner call in a circuit-breaker so that repeatedly
+   broken topics don't choke throughput
+   ([#373](https://github.com/Shopify/sarama/pull/373)).
+
+Bug Fixes:
+ - Fix the producer's internal reference counting in certain unusual scenarios
+   ([#367](https://github.com/Shopify/sarama/pull/367)).
+ - Fix the consumer's internal reference counting in certain unusual scenarios
+   ([#369](https://github.com/Shopify/sarama/pull/369)).
+ - Fix a condition where the producer's internal control messages could have
+   gotten stuck ([#368](https://github.com/Shopify/sarama/pull/368)).
+ - Fix an issue where invalid partition lists would be cached when asking for
+   metadata for a non-existant topic ([#372](https://github.com/Shopify/sarama/pull/372)).
+
+
+#### Version 1.0.0 (2015-03-17)
+
+Version 1.0.0 is the first tagged version, and is almost a complete rewrite. The primary differences with previous untagged versions are:
+
+- The producer has been rewritten; there is now a `SyncProducer` with a blocking API, and an `AsyncProducer` that is non-blocking.
+- The consumer has been rewritten to only open one connection per broker instead of one connection per partition.
+- The main types of Sarama are now interfaces to make depedency injection easy; mock implementations for `Consumer`, `SyncProducer` and `AsyncProducer` are provided in the `github.com/Shopify/sarama/mocks` package.
+- For most uses cases, it is no longer necessary to open a `Client`; this will be done for you.
+- All the configuration values have been unified in the `Config` struct.
+- Much improved test suite.
@@ -0,0 +1,20 @@
+Copyright (c) 2013 Evan Huus
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
@@ -0,0 +1,21 @@
+default: fmt vet errcheck test
+
+test:
+	go test -v -timeout 60s -race ./...
+
+vet:
+	go vet ./...
+
+errcheck:
+	errcheck github.com/Shopify/sarama/...
+
+fmt:
+	@if [ -n "$$(go fmt ./...)" ]; then echo 'Please run go fmt on your code.' && exit 1; fi
+
+install_dependencies: install_errcheck get
+
+install_errcheck:
+	go get github.com/kisielk/errcheck
+
+get:
+	go get -t
@@ -0,0 +1,36 @@
+sarama
+======
+
+[![GoDoc](https://godoc.org/github.com/Shopify/sarama?status.png)](https://godoc.org/github.com/Shopify/sarama)
+[![Build Status](https://travis-ci.org/Shopify/sarama.svg?branch=master)](https://travis-ci.org/Shopify/sarama)
+
+Sarama is an MIT-licensed Go client library for [Apache Kafka](https://kafka.apache.org/) version 0.8 (and later).
+
+### Getting started
+
+- API documentation and examples are available via [godoc](https://godoc.org/github.com/Shopify/sarama).
+- Mocks for testing are available in the [mocks](./mocks) subpackage.
+- The [examples](./examples) directory contains more elaborate example applications.
+- The [tools](./tools) directory contains command line tools that can be useful for testing, diagnostics, and instrumentation.
+
+### Compatibility and API stability
+
+Sarama provides a "2 releases + 2 months" compatibility guarantee: we support
+the two latest stable releases of Kafka and Go, and we provide a two month
+grace period for older releases. This means we currently officially support
+Go 1.7 and 1.6, and Kafka 0.10.0 and 0.9.0, although older releases are
+still likely to work.
+
+Sarama follows semantic versioning and provides API stability via the gopkg.in service.
+You can import a version with a guaranteed stable API via http://gopkg.in/Shopify/sarama.v1.
+A changelog is available [here](CHANGELOG.md).
+
+### Contributing
+
+* Get started by checking our [contribution guidelines](https://github.com/Shopify/sarama/blob/master/CONTRIBUTING.md).
+* Read the [Sarama wiki](https://github.com/Shopify/sarama/wiki) for more
+  technical and design details.
+* The [Kafka Protocol Specification](https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol)
+  contains a wealth of useful information.
+* For more general issues, there is [a google group](https://groups.google.com/forum/#!forum/kafka-clients) for Kafka client developers.
+* If you have any questions, just ask!
@@ -0,0 +1,20 @@
+# -*- mode: ruby -*-
+# vi: set ft=ruby :
+
+# Vagrantfile API/syntax version. Don't touch unless you know what you're doing!
+VAGRANTFILE_API_VERSION = "2"
+
+# We have 5 * 192MB ZK processes and 5 * 320MB Kafka processes => 2560MB
+MEMORY = 3072
+
+Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
+  config.vm.box = "ubuntu/trusty64"
+
+  config.vm.provision :shell, path: "vagrant/provision.sh"
+
+  config.vm.network "private_network", ip: "192.168.100.67"
+
+  config.vm.provider "virtualbox" do |v|
+    v.memory = MEMORY
+  end
+end
@@ -0,0 +1,24 @@
+package sarama
+
+type ApiVersionsRequest struct {
+}
+
+func (r *ApiVersionsRequest) encode(pe packetEncoder) error {
+	return nil
+}
+
+func (r *ApiVersionsRequest) decode(pd packetDecoder, version int16) (err error) {
+	return nil
+}
+
+func (r *ApiVersionsRequest) key() int16 {
+	return 18
+}
+
+func (r *ApiVersionsRequest) version() int16 {
+	return 0
+}
+
+func (r *ApiVersionsRequest) requiredVersion() KafkaVersion {
+	return V0_10_0_0
+}
@@ -0,0 +1,86 @@
+package sarama
+
+type ApiVersionsResponseBlock struct {
+	ApiKey     int16
+	MinVersion int16
+	MaxVersion int16
+}
+
+func (b *ApiVersionsResponseBlock) encode(pe packetEncoder) error {
+	pe.putInt16(b.ApiKey)
+	pe.putInt16(b.MinVersion)
+	pe.putInt16(b.MaxVersion)
+	return nil
+}
+
+func (b *ApiVersionsResponseBlock) decode(pd packetDecoder) error {
+	var err error
+
+	if b.ApiKey, err = pd.getInt16(); err != nil {
+		return err
+	}
+
+	if b.MinVersion, err = pd.getInt16(); err != nil {
+		return err
+	}
+
+	if b.MaxVersion, err = pd.getInt16(); err != nil {
+		return err
+	}
+
+	return nil
+}
+
+type ApiVersionsResponse struct {
+	Err         KError
+	ApiVersions []*ApiVersionsResponseBlock
+}
+
+func (r *ApiVersionsResponse) encode(pe packetEncoder) error {
+	pe.putInt16(int16(r.Err))
+	if err := pe.putArrayLength(len(r.ApiVersions)); err != nil {
+		return err
+	}
+	for _, apiVersion := range r.ApiVersions {
+		if err := apiVersion.encode(pe); err != nil {
+			return err
+		}
+	}
+	return nil
+}
+
+func (r *ApiVersionsResponse) decode(pd packetDecoder, version int16) error {
+	if kerr, err := pd.getInt16(); err != nil {
+		return err
+	} else {
+		r.Err = KError(kerr)
+	}
+
+	numBlocks, err := pd.getArrayLength()
+	if err != nil {
+		return err
+	}
+
+	r.ApiVersions = make([]*ApiVersionsResponseBlock, numBlocks)
+	for i := 0; i < numBlocks; i++ {
+		block := new(ApiVersionsResponseBlock)
+		if err := block.decode(pd); err != nil {
+			return err
+		}
+		r.ApiVersions[i] = block
+	}
+
+	return nil
+}
+
+func (r *ApiVersionsResponse) key() int16 {
+	return 18
+}
+
+func (r *ApiVersionsResponse) version() int16 {
+	return 0
+}
+
+func (r *ApiVersionsResponse) requiredVersion() KafkaVersion {
+	return V0_10_0_0
+}
@@ -0,0 +1,905 @@
+package sarama
+
+import (
+	"fmt"
+	"sync"
+	"time"
+
+	"github.com/eapache/go-resiliency/breaker"
+	"github.com/eapache/queue"
+)
+
+// AsyncProducer publishes Kafka messages using a non-blocking API. It routes messages
+// to the correct broker for the provided topic-partition, refreshing metadata as appropriate,
+// and parses responses for errors. You must read from the Errors() channel or the
+// producer will deadlock. You must call Close() or AsyncClose() on a producer to avoid
+// leaks: it will not be garbage-collected automatically when it passes out of
+// scope.
+type AsyncProducer interface {
+
+	// AsyncClose triggers a shutdown of the producer, flushing any messages it may
+	// have buffered. The shutdown has completed when both the Errors and Successes
+	// channels have been closed. When calling AsyncClose, you *must* continue to
+	// read from those channels in order to drain the results of any messages in
+	// flight.
+	AsyncClose()
+
+	// Close shuts down the producer and flushes any messages it may have buffered.
+	// You must call this function before a producer object passes out of scope, as
+	// it may otherwise leak memory. You must call this before calling Close on the
+	// underlying client.
+	Close() error
+
+	// Input is the input channel for the user to write messages to that they
+	// wish to send.
+	Input() chan<- *ProducerMessage
+
+	// Successes is the success output channel back to the user when AckSuccesses is
+	// enabled. If Return.Successes is true, you MUST read from this channel or the
+	// Producer will deadlock. It is suggested that you send and read messages
+	// together in a single select statement.
+	Successes() <-chan *ProducerMessage
+
+	// Errors is the error output channel back to the user. You MUST read from this
+	// channel or the Producer will deadlock when the channel is full. Alternatively,
+	// you can set Producer.Return.Errors in your config to false, which prevents
+	// errors to be returned.
+	Errors() <-chan *ProducerError
+}
+
+type asyncProducer struct {
+	client    Client
+	conf      *Config
+	ownClient bool
+
+	errors                    chan *ProducerError
+	input, successes, retries chan *ProducerMessage
+	inFlight                  sync.WaitGroup
+
+	brokers    map[*Broker]chan<- *ProducerMessage
+	brokerRefs map[chan<- *ProducerMessage]int
+	brokerLock sync.Mutex
+}
+
+// NewAsyncProducer creates a new AsyncProducer using the given broker addresses and configuration.
+func NewAsyncProducer(addrs []string, conf *Config) (AsyncProducer, error) {
+	client, err := NewClient(addrs, conf)
+	if err != nil {
+		return nil, err
+	}
+
+	p, err := NewAsyncProducerFromClient(client)
+	if err != nil {
+		return nil, err
+	}
+	p.(*asyncProducer).ownClient = true
+	return p, nil
+}
+
+// NewAsyncProducerFromClient creates a new Producer using the given client. It is still
+// necessary to call Close() on the underlying client when shutting down this producer.
+func NewAsyncProducerFromClient(client Client) (AsyncProducer, error) {
+	// Check that we are not dealing with a closed Client before processing any other arguments
+	if client.Closed() {
+		return nil, ErrClosedClient
+	}
+
+	p := &asyncProducer{
+		client:     client,
+		conf:       client.Config(),
+		errors:     make(chan *ProducerError),
+		input:      make(chan *ProducerMessage),
+		successes:  make(chan *ProducerMessage),
+		retries:    make(chan *ProducerMessage),
+		brokers:    make(map[*Broker]chan<- *ProducerMessage),
+		brokerRefs: make(map[chan<- *ProducerMessage]int),
+	}
+
+	// launch our singleton dispatchers
+	go withRecover(p.dispatcher)
+	go withRecover(p.retryHandler)
+
+	return p, nil
+}
+
+type flagSet int8
+
+const (
+	syn      flagSet = 1 << iota // first message from partitionProducer to brokerProducer
+	fin                          // final message from partitionProducer to brokerProducer and back
+	shutdown                     // start the shutdown process
+)
+
+// ProducerMessage is the collection of elements passed to the Producer in order to send a message.
+type ProducerMessage struct {
+	Topic string // The Kafka topic for this message.
+	// The partitioning key for this message. Pre-existing Encoders include
+	// StringEncoder and ByteEncoder.
+	Key Encoder
+	// The actual message to store in Kafka. Pre-existing Encoders include
+	// StringEncoder and ByteEncoder.
+	Value Encoder
+
+	// This field is used to hold arbitrary data you wish to include so it
+	// will be available when receiving on the Successes and Errors channels.
+	// Sarama completely ignores this field and is only to be used for
+	// pass-through data.
+	Metadata interface{}
+
+	// Below this point are filled in by the producer as the message is processed
+
+	// Offset is the offset of the message stored on the broker. This is only
+	// guaranteed to be defined if the message was successfully delivered and
+	// RequiredAcks is not NoResponse.
+	Offset int64
+	// Partition is the partition that the message was sent to. This is only
+	// guaranteed to be defined if the message was successfully delivered.
+	Partition int32
+	// Timestamp is the timestamp assigned to the message by the broker. This
+	// is only guaranteed to be defined if the message was successfully
+	// delivered, RequiredAcks is not NoResponse, and the Kafka broker is at
+	// least version 0.10.0.
+	Timestamp time.Time
+
+	retries int
+	flags   flagSet
+}
+
+const producerMessageOverhead = 26 // the metadata overhead of CRC, flags, etc.
+
+func (m *ProducerMessage) byteSize() int {
+	size := producerMessageOverhead
+	if m.Key != nil {
+		size += m.Key.Length()
+	}
+	if m.Value != nil {
+		size += m.Value.Length()
+	}
+	return size
+}
+
+func (m *ProducerMessage) clear() {
+	m.flags = 0
+	m.retries = 0
+}
+
+// ProducerError is the type of error generated when the producer fails to deliver a message.
+// It contains the original ProducerMessage as well as the actual error value.
+type ProducerError struct {
+	Msg *ProducerMessage
+	Err error
+}
+
+func (pe ProducerError) Error() string {
+	return fmt.Sprintf("kafka: Failed to produce message to topic %s: %s", pe.Msg.Topic, pe.Err)
+}
+
+// ProducerErrors is a type that wraps a batch of "ProducerError"s and implements the Error interface.
+// It can be returned from the Producer's Close method to avoid the need to manually drain the Errors channel
+// when closing a producer.
+type ProducerErrors []*ProducerError
+
+func (pe ProducerErrors) Error() string {
+	return fmt.Sprintf("kafka: Failed to deliver %d messages.", len(pe))
+}
+
+func (p *asyncProducer) Errors() <-chan *ProducerError {
+	return p.errors
+}
+
+func (p *asyncProducer) Successes() <-chan *ProducerMessage {
+	return p.successes
+}
+
+func (p *asyncProducer) Input() chan<- *ProducerMessage {
+	return p.input
+}
+
+func (p *asyncProducer) Close() error {
+	p.AsyncClose()
+
+	if p.conf.Producer.Return.Successes {
+		go withRecover(func() {
+			for _ = range p.successes {
+			}
+		})
+	}
+
+	var errors ProducerErrors
+	if p.conf.Producer.Return.Errors {
+		for event := range p.errors {
+			errors = append(errors, event)
+		}
+	} else {
+		<-p.errors
+	}
+
+	if len(errors) > 0 {
+		return errors
+	}
+	return nil
+}
+
+func (p *asyncProducer) AsyncClose() {
+	go withRecover(p.shutdown)
+}
+
+// singleton
+// dispatches messages by topic
+func (p *asyncProducer) dispatcher() {
+	handlers := make(map[string]chan<- *ProducerMessage)
+	shuttingDown := false
+
+	for msg := range p.input {
+		if msg == nil {
+			Logger.Println("Something tried to send a nil message, it was ignored.")
+			continue
+		}
+
+		if msg.flags&shutdown != 0 {
+			shuttingDown = true
+			p.inFlight.Done()
+			continue
+		} else if msg.retries == 0 {
+			if shuttingDown {
+				// we can't just call returnError here because that decrements the wait group,
+				// which hasn't been incremented yet for this message, and shouldn't be
+				pErr := &ProducerError{Msg: msg, Err: ErrShuttingDown}
+				if p.conf.Producer.Return.Errors {
+					p.errors <- pErr
+				} else {
+					Logger.Println(pErr)
+				}
+				continue
+			}
+			p.inFlight.Add(1)
+		}
+
+		if msg.byteSize() > p.conf.Producer.MaxMessageBytes {
+			p.returnError(msg, ErrMessageSizeTooLarge)
+			continue
+		}
+
+		handler := handlers[msg.Topic]
+		if handler == nil {
+			handler = p.newTopicProducer(msg.Topic)
+			handlers[msg.Topic] = handler
+		}
+
+		handler <- msg
+	}
+
+	for _, handler := range handlers {
+		close(handler)
+	}
+}
+
+// one per topic
+// partitions messages, then dispatches them by partition
+type topicProducer struct {
+	parent *asyncProducer
+	topic  string
+	input  <-chan *ProducerMessage
+
+	breaker     *breaker.Breaker
+	handlers    map[int32]chan<- *ProducerMessage
+	partitioner Partitioner
+}
+
+func (p *asyncProducer) newTopicProducer(topic string) chan<- *ProducerMessage {
+	input := make(chan *ProducerMessage, p.conf.ChannelBufferSize)
+	tp := &topicProducer{
+		parent:      p,
+		topic:       topic,
+		input:       input,
+		breaker:     breaker.New(3, 1, 10*time.Second),
+		handlers:    make(map[int32]chan<- *ProducerMessage),
+		partitioner: p.conf.Producer.Partitioner(topic),
+	}
+	go withRecover(tp.dispatch)
+	return input
+}
+
+func (tp *topicProducer) dispatch() {
+	for msg := range tp.input {
+		if msg.retries == 0 {
+			if err := tp.partitionMessage(msg); err != nil {
+				tp.parent.returnError(msg, err)
+				continue
+			}
+		}
+
+		handler := tp.handlers[msg.Partition]
+		if handler == nil {
+			handler = tp.parent.newPartitionProducer(msg.Topic, msg.Partition)
+			tp.handlers[msg.Partition] = handler
+		}
+
+		handler <- msg
+	}
+
+	for _, handler := range tp.handlers {
+		close(handler)
+	}
+}
+
+func (tp *topicProducer) partitionMessage(msg *ProducerMessage) error {
+	var partitions []int32
+
+	err := tp.breaker.Run(func() (err error) {
+		if tp.partitioner.RequiresConsistency() {
+			partitions, err = tp.parent.client.Partitions(msg.Topic)
+		} else {
+			partitions, err = tp.parent.client.WritablePartitions(msg.Topic)
+		}
+		return
+	})
+
+	if err != nil {
+		return err
+	}
+
+	numPartitions := int32(len(partitions))
+
+	if numPartitions == 0 {
+		return ErrLeaderNotAvailable
+	}
+
+	choice, err := tp.partitioner.Partition(msg, numPartitions)
+
+	if err != nil {
+		return err
+	} else if choice < 0 || choice >= numPartitions {
+		return ErrInvalidPartition
+	}
+
+	msg.Partition = partitions[choice]
+
+	return nil
+}
+
+// one per partition per topic
+// dispatches messages to the appropriate broker
+// also responsible for maintaining message order during retries
+type partitionProducer struct {
+	parent    *asyncProducer
+	topic     string
+	partition int32
+	input     <-chan *ProducerMessage
+
+	leader  *Broker
+	breaker *breaker.Breaker
+	output  chan<- *ProducerMessage
+
+	// highWatermark tracks the "current" retry level, which is the only one where we actually let messages through,
+	// all other messages get buffered in retryState[msg.retries].buf to preserve ordering
+	// retryState[msg.retries].expectChaser simply tracks whether we've seen a fin message for a given level (and
+	// therefore whether our buffer is complete and safe to flush)
+	highWatermark int
+	retryState    []partitionRetryState
+}
+
+type partitionRetryState struct {
+	buf          []*ProducerMessage
+	expectChaser bool
+}
+
+func (p *asyncProducer) newPartitionProducer(topic string, partition int32) chan<- *ProducerMessage {
+	input := make(chan *ProducerMessage, p.conf.ChannelBufferSize)
+	pp := &partitionProducer{
+		parent:    p,
+		topic:     topic,
+		partition: partition,
+		input:     input,
+
+		breaker:    breaker.New(3, 1, 10*time.Second),
+		retryState: make([]partitionRetryState, p.conf.Producer.Retry.Max+1),
+	}
+	go withRecover(pp.dispatch)
+	return input
+}
+
+func (pp *partitionProducer) dispatch() {
+	// try to prefetch the leader; if this doesn't work, we'll do a proper call to `updateLeader`
+	// on the first message
+	pp.leader, _ = pp.parent.client.Leader(pp.topic, pp.partition)
+	if pp.leader != nil {
+		pp.output = pp.parent.getBrokerProducer(pp.leader)
+		pp.parent.inFlight.Add(1) // we're generating a syn message; track it so we don't shut down while it's still inflight
+		pp.output <- &ProducerMessage{Topic: pp.topic, Partition: pp.partition, flags: syn}
+	}
+
+	for msg := range pp.input {
+		if msg.retries > pp.highWatermark {
+			// a new, higher, retry level; handle it and then back off
+			pp.newHighWatermark(msg.retries)
+			time.Sleep(pp.parent.conf.Producer.Retry.Backoff)
+		} else if pp.highWatermark > 0 {
+			// we are retrying something (else highWatermark would be 0) but this message is not a *new* retry level
+			if msg.retries < pp.highWatermark {
+				// in fact this message is not even the current retry level, so buffer it for now (unless it's a just a fin)
+				if msg.flags&fin == fin {
+					pp.retryState[msg.retries].expectChaser = false
+					pp.parent.inFlight.Done() // this fin is now handled and will be garbage collected
+				} else {
+					pp.retryState[msg.retries].buf = append(pp.retryState[msg.retries].buf, msg)
+				}
+				continue
+			} else if msg.flags&fin == fin {
+				// this message is of the current retry level (msg.retries == highWatermark) and the fin flag is set,
+				// meaning this retry level is done and we can go down (at least) one level and flush that
+				pp.retryState[pp.highWatermark].expectChaser = false
+				pp.flushRetryBuffers()
+				pp.parent.inFlight.Done() // this fin is now handled and will be garbage collected
+				continue
+			}
+		}
+
+		// if we made it this far then the current msg contains real data, and can be sent to the next goroutine
+		// without breaking any of our ordering guarantees
+
+		if pp.output == nil {
+			if err := pp.updateLeader(); err != nil {
+				pp.parent.returnError(msg, err)
+				time.Sleep(pp.parent.conf.Producer.Retry.Backoff)
+				continue
+			}
+			Logger.Printf("producer/leader/%s/%d selected broker %d\n", pp.topic, pp.partition, pp.leader.ID())
+		}
+
+		pp.output <- msg
+	}
+
+	if pp.output != nil {
+		pp.parent.unrefBrokerProducer(pp.leader, pp.output)
+	}
+}
+
+func (pp *partitionProducer) newHighWatermark(hwm int) {
+	Logger.Printf("producer/leader/%s/%d state change to [retrying-%d]\n", pp.topic, pp.partition, hwm)
+	pp.highWatermark = hwm
+
+	// send off a fin so that we know when everything "in between" has made it
+	// back to us and we can safely flush the backlog (otherwise we risk re-ordering messages)
+	pp.retryState[pp.highWatermark].expectChaser = true
+	pp.parent.inFlight.Add(1) // we're generating a fin message; track it so we don't shut down while it's still inflight
+	pp.output <- &ProducerMessage{Topic: pp.topic, Partition: pp.partition, flags: fin, retries: pp.highWatermark - 1}
+
+	// a new HWM means that our current broker selection is out of date
+	Logger.Printf("producer/leader/%s/%d abandoning broker %d\n", pp.topic, pp.partition, pp.leader.ID())
+	pp.parent.unrefBrokerProducer(pp.leader, pp.output)
+	pp.output = nil
+}
+
+func (pp *partitionProducer) flushRetryBuffers() {
+	Logger.Printf("producer/leader/%s/%d state change to [flushing-%d]\n", pp.topic, pp.partition, pp.highWatermark)
+	for {
+		pp.highWatermark--
+
+		if pp.output == nil {
+			if err := pp.updateLeader(); err != nil {
+				pp.parent.returnErrors(pp.retryState[pp.highWatermark].buf, err)
+				goto flushDone
+			}
+			Logger.Printf("producer/leader/%s/%d selected broker %d\n", pp.topic, pp.partition, pp.leader.ID())
+		}
+
+		for _, msg := range pp.retryState[pp.highWatermark].buf {
+			pp.output <- msg
+		}
+
+	flushDone:
+		pp.retryState[pp.highWatermark].buf = nil
+		if pp.retryState[pp.highWatermark].expectChaser {
+			Logger.Printf("producer/leader/%s/%d state change to [retrying-%d]\n", pp.topic, pp.partition, pp.highWatermark)
+			break
+		} else if pp.highWatermark == 0 {
+			Logger.Printf("producer/leader/%s/%d state change to [normal]\n", pp.topic, pp.partition)
+			break
+		}
+	}
+}
+
+func (pp *partitionProducer) updateLeader() error {
+	return pp.breaker.Run(func() (err error) {
+		if err = pp.parent.client.RefreshMetadata(pp.topic); err != nil {
+			return err
+		}
+
+		if pp.leader, err = pp.parent.client.Leader(pp.topic, pp.partition); err != nil {
+			return err
+		}
+
+		pp.output = pp.parent.getBrokerProducer(pp.leader)
+		pp.parent.inFlight.Add(1) // we're generating a syn message; track it so we don't shut down while it's still inflight
+		pp.output <- &ProducerMessage{Topic: pp.topic, Partition: pp.partition, flags: syn}
+
+		return nil
+	})
+}
+
+// one per broker; also constructs an associated flusher
+func (p *asyncProducer) newBrokerProducer(broker *Broker) chan<- *ProducerMessage {
+	var (
+		input     = make(chan *ProducerMessage)
+		bridge    = make(chan *produceSet)
+		responses = make(chan *brokerProducerResponse)
+	)
+
+	bp := &brokerProducer{
+		parent:         p,
+		broker:         broker,
+		input:          input,
+		output:         bridge,
+		responses:      responses,
+		buffer:         newProduceSet(p),
+		currentRetries: make(map[string]map[int32]error),
+	}
+	go withRecover(bp.run)
+
+	// minimal bridge to make the network response `select`able
+	go withRecover(func() {
+		for set := range bridge {
+			request := set.buildRequest()
+
+			response, err := broker.Produce(request)
+
+			responses <- &brokerProducerResponse{
+				set: set,
+				err: err,
+				res: response,
+			}
+		}
+		close(responses)
+	})
+
+	return input
+}
+
+type brokerProducerResponse struct {
+	set *produceSet
+	err error
+	res *ProduceResponse
+}
+
+// groups messages together into appropriately-sized batches for sending to the broker
+// handles state related to retries etc
+type brokerProducer struct {
+	parent *asyncProducer
+	broker *Broker
+
+	input     <-chan *ProducerMessage
+	output    chan<- *produceSet
+	responses <-chan *brokerProducerResponse
+
+	buffer     *produceSet
+	timer      <-chan time.Time
+	timerFired bool
+
+	closing        error
+	currentRetries map[string]map[int32]error
+}
+
+func (bp *brokerProducer) run() {
+	var output chan<- *produceSet
+	Logger.Printf("producer/broker/%d starting up\n", bp.broker.ID())
+
+	for {
+		select {
+		case msg := <-bp.input:
+			if msg == nil {
+				bp.shutdown()
+				return
+			}
+
+			if msg.flags&syn == syn {
+				Logger.Printf("producer/broker/%d state change to [open] on %s/%d\n",
+					bp.broker.ID(), msg.Topic, msg.Partition)
+				if bp.currentRetries[msg.Topic] == nil {
+					bp.currentRetries[msg.Topic] = make(map[int32]error)
+				}
+				bp.currentRetries[msg.Topic][msg.Partition] = nil
+				bp.parent.inFlight.Done()
+				continue
+			}
+
+			if reason := bp.needsRetry(msg); reason != nil {
+				bp.parent.retryMessage(msg, reason)
+
+				if bp.closing == nil && msg.flags&fin == fin {
+					// we were retrying this partition but we can start processing again
+					delete(bp.currentRetries[msg.Topic], msg.Partition)
+					Logger.Printf("producer/broker/%d state change to [closed] on %s/%d\n",
+						bp.broker.ID(), msg.Topic, msg.Partition)
+				}
+
+				continue
+			}
+
+			if bp.buffer.wouldOverflow(msg) {
+				if err := bp.waitForSpace(msg); err != nil {
+					bp.parent.retryMessage(msg, err)
+					continue
+				}
+			}
+
+			if err := bp.buffer.add(msg); err != nil {
+				bp.parent.returnError(msg, err)
+				continue
+			}
+
+			if bp.parent.conf.Producer.Flush.Frequency > 0 && bp.timer == nil {
+				bp.timer = time.After(bp.parent.conf.Producer.Flush.Frequency)
+			}
+		case <-bp.timer:
+			bp.timerFired = true
+		case output <- bp.buffer:
+			bp.rollOver()
+		case response := <-bp.responses:
+			bp.handleResponse(response)
+		}
+
+		if bp.timerFired || bp.buffer.readyToFlush() {
+			output = bp.output
+		} else {
+			output = nil
+		}
+	}
+}
+
+func (bp *brokerProducer) shutdown() {
+	for !bp.buffer.empty() {
+		select {
+		case response := <-bp.responses:
+			bp.handleResponse(response)
+		case bp.output <- bp.buffer:
+			bp.rollOver()
+		}
+	}
+	close(bp.output)
+	for response := range bp.responses {
+		bp.handleResponse(response)
+	}
+
+	Logger.Printf("producer/broker/%d shut down\n", bp.broker.ID())
+}
+
+func (bp *brokerProducer) needsRetry(msg *ProducerMessage) error {
+	if bp.closing != nil {
+		return bp.closing
+	}
+
+	return bp.currentRetries[msg.Topic][msg.Partition]
+}
+
+func (bp *brokerProducer) waitForSpace(msg *ProducerMessage) error {
+	Logger.Printf("producer/broker/%d maximum request accumulated, waiting for space\n", bp.broker.ID())
+
+	for {
+		select {
+		case response := <-bp.responses:
+			bp.handleResponse(response)
+			// handling a response can change our state, so re-check some things
+			if reason := bp.needsRetry(msg); reason != nil {
+				return reason
+			} else if !bp.buffer.wouldOverflow(msg) {
+				return nil
+			}
+		case bp.output <- bp.buffer:
+			bp.rollOver()
+			return nil
+		}
+	}
+}
+
+func (bp *brokerProducer) rollOver() {
+	bp.timer = nil
+	bp.timerFired = false
+	bp.buffer = newProduceSet(bp.parent)
+}
+
+func (bp *brokerProducer) handleResponse(response *brokerProducerResponse) {
+	if response.err != nil {
+		bp.handleError(response.set, response.err)
+	} else {
+		bp.handleSuccess(response.set, response.res)
+	}
+
+	if bp.buffer.empty() {
+		bp.rollOver() // this can happen if the response invalidated our buffer
+	}
+}
+
+func (bp *brokerProducer) handleSuccess(sent *produceSet, response *ProduceResponse) {
+	// we iterate through the blocks in the request set, not the response, so that we notice
+	// if the response is missing a block completely
+	sent.eachPartition(func(topic string, partition int32, msgs []*ProducerMessage) {
+		if response == nil {
+			// this only happens when RequiredAcks is NoResponse, so we have to assume success
+			bp.parent.returnSuccesses(msgs)
+			return
+		}
+
+		block := response.GetBlock(topic, partition)
+		if block == nil {
+			bp.parent.returnErrors(msgs, ErrIncompleteResponse)
+			return
+		}
+
+		switch block.Err {
+		// Success
+		case ErrNoError:
+			if bp.parent.conf.Version.IsAtLeast(V0_10_0_0) && !block.Timestamp.IsZero() {
+				for _, msg := range msgs {
+					msg.Timestamp = block.Timestamp
+				}
+			}
+			for i, msg := range msgs {
+				msg.Offset = block.Offset + int64(i)
+			}
+			bp.parent.returnSuccesses(msgs)
+		// Retriable errors
+		case ErrInvalidMessage, ErrUnknownTopicOrPartition, ErrLeaderNotAvailable, ErrNotLeaderForPartition,
+			ErrRequestTimedOut, ErrNotEnoughReplicas, ErrNotEnoughReplicasAfterAppend:
+			Logger.Printf("producer/broker/%d state change to [retrying] on %s/%d because %v\n",
+				bp.broker.ID(), topic, partition, block.Err)
+			bp.currentRetries[topic][partition] = block.Err
+			bp.parent.retryMessages(msgs, block.Err)
+			bp.parent.retryMessages(bp.buffer.dropPartition(topic, partition), block.Err)
+		// Other non-retriable errors
+		default:
+			bp.parent.returnErrors(msgs, block.Err)
+		}
+	})
+}
+
+func (bp *brokerProducer) handleError(sent *produceSet, err error) {
+	switch err.(type) {
+	case PacketEncodingError:
+		sent.eachPartition(func(topic string, partition int32, msgs []*ProducerMessage) {
+			bp.parent.returnErrors(msgs, err)
+		})
+	default:
+		Logger.Printf("producer/broker/%d state change to [closing] because %s\n", bp.broker.ID(), err)
+		bp.parent.abandonBrokerConnection(bp.broker)
+		_ = bp.broker.Close()
+		bp.closing = err
+		sent.eachPartition(func(topic string, partition int32, msgs []*ProducerMessage) {
+			bp.parent.retryMessages(msgs, err)
+		})
+		bp.buffer.eachPartition(func(topic string, partition int32, msgs []*ProducerMessage) {
+			bp.parent.retryMessages(msgs, err)
+		})
+		bp.rollOver()
+	}
+}
+
+// singleton
+// effectively a "bridge" between the flushers and the dispatcher in order to avoid deadlock
+// based on https://godoc.org/github.com/eapache/channels#InfiniteChannel
+func (p *asyncProducer) retryHandler() {
+	var msg *ProducerMessage
+	buf := queue.New()
+
+	for {
+		if buf.Length() == 0 {
+			msg = <-p.retries
+		} else {
+			select {
+			case msg = <-p.retries:
+			case p.input <- buf.Peek().(*ProducerMessage):
+				buf.Remove()
+				continue
+			}
+		}
+
+		if msg == nil {
+			return
+		}
+
+		buf.Add(msg)
+	}
+}
+
+// utility functions
+
+func (p *asyncProducer) shutdown() {
+	Logger.Println("Producer shutting down.")
+	p.inFlight.Add(1)
+	p.input <- &ProducerMessage{flags: shutdown}
+
+	p.inFlight.Wait()
+
+	if p.ownClient {
+		err := p.client.Close()
+		if err != nil {
+			Logger.Println("producer/shutdown failed to close the embedded client:", err)
+		}
+	}
+
+	close(p.input)
+	close(p.retries)
+	close(p.errors)
+	close(p.successes)
+}
+
+func (p *asyncProducer) returnError(msg *ProducerMessage, err error) {
+	msg.clear()
+	pErr := &ProducerError{Msg: msg, Err: err}
+	if p.conf.Producer.Return.Errors {
+		p.errors <- pErr
+	} else {
+		Logger.Println(pErr)
+	}
+	p.inFlight.Done()
+}
+
+func (p *asyncProducer) returnErrors(batch []*ProducerMessage, err error) {
+	for _, msg := range batch {
+		p.returnError(msg, err)
+	}
+}
+
+func (p *asyncProducer) returnSuccesses(batch []*ProducerMessage) {
+	for _, msg := range batch {
+		if p.conf.Producer.Return.Successes {
+			msg.clear()
+			p.successes <- msg
+		}
+		p.inFlight.Done()
+	}
+}
+
+func (p *asyncProducer) retryMessage(msg *ProducerMessage, err error) {
+	if msg.retries >= p.conf.Producer.Retry.Max {
+		p.returnError(msg, err)
+	} else {
+		msg.retries++
+		p.retries <- msg
+	}
+}
+
+func (p *asyncProducer) retryMessages(batch []*ProducerMessage, err error) {
+	for _, msg := range batch {
+		p.retryMessage(msg, err)
+	}
+}
+
+func (p *asyncProducer) getBrokerProducer(broker *Broker) chan<- *ProducerMessage {
+	p.brokerLock.Lock()
+	defer p.brokerLock.Unlock()
+
+	bp := p.brokers[broker]
+
+	if bp == nil {
+		bp = p.newBrokerProducer(broker)
+		p.brokers[broker] = bp
+		p.brokerRefs[bp] = 0
+	}
+
+	p.brokerRefs[bp]++
+
+	return bp
+}
+
+func (p *asyncProducer) unrefBrokerProducer(broker *Broker, bp chan<- *ProducerMessage) {
+	p.brokerLock.Lock()
+	defer p.brokerLock.Unlock()
+
+	p.brokerRefs[bp]--
+	if p.brokerRefs[bp] == 0 {
+		close(bp)
+		delete(p.brokerRefs, bp)
+
+		if p.brokers[broker] == bp {
+			delete(p.brokers, broker)
+		}
+	}
+}
+
+func (p *asyncProducer) abandonBrokerConnection(broker *Broker) {
+	p.brokerLock.Lock()
+	defer p.brokerLock.Unlock()
+
+	delete(p.brokers, broker)
+}
@@ -0,0 +1,674 @@
+package sarama
+
+import (
+	"crypto/tls"
+	"encoding/binary"
+	"fmt"
+	"io"
+	"net"
+	"strconv"
+	"sync"
+	"sync/atomic"
+	"time"
+
+	"github.com/rcrowley/go-metrics"
+)
+
+// Broker represents a single Kafka broker connection. All operations on this object are entirely concurrency-safe.
+type Broker struct {
+	id   int32
+	addr string
+
+	conf          *Config
+	correlationID int32
+	conn          net.Conn
+	connErr       error
+	lock          sync.Mutex
+	opened        int32
+
+	responses chan responsePromise
+	done      chan bool
+
+	incomingByteRate       metrics.Meter
+	requestRate            metrics.Meter
+	requestSize            metrics.Histogram
+	requestLatency         metrics.Histogram
+	outgoingByteRate       metrics.Meter
+	responseRate           metrics.Meter
+	responseSize           metrics.Histogram
+	brokerIncomingByteRate metrics.Meter
+	brokerRequestRate      metrics.Meter
+	brokerRequestSize      metrics.Histogram
+	brokerRequestLatency   metrics.Histogram
+	brokerOutgoingByteRate metrics.Meter
+	brokerResponseRate     metrics.Meter
+	brokerResponseSize     metrics.Histogram
+}
+
+type responsePromise struct {
+	requestTime   time.Time
+	correlationID int32
+	packets       chan []byte
+	errors        chan error
+}
+
+// NewBroker creates and returns a Broker targetting the given host:port address.
+// This does not attempt to actually connect, you have to call Open() for that.
+func NewBroker(addr string) *Broker {
+	return &Broker{id: -1, addr: addr}
+}
+
+// Open tries to connect to the Broker if it is not already connected or connecting, but does not block
+// waiting for the connection to complete. This means that any subsequent operations on the broker will
+// block waiting for the connection to succeed or fail. To get the effect of a fully synchronous Open call,
+// follow it by a call to Connected(). The only errors Open will return directly are ConfigurationError or
+// AlreadyConnected. If conf is nil, the result of NewConfig() is used.
+func (b *Broker) Open(conf *Config) error {
+	if !atomic.CompareAndSwapInt32(&b.opened, 0, 1) {
+		return ErrAlreadyConnected
+	}
+
+	if conf == nil {
+		conf = NewConfig()
+	}
+
+	err := conf.Validate()
+	if err != nil {
+		return err
+	}
+
+	b.lock.Lock()
+
+	go withRecover(func() {
+		defer b.lock.Unlock()
+
+		dialer := net.Dialer{
+			Timeout:   conf.Net.DialTimeout,
+			KeepAlive: conf.Net.KeepAlive,
+		}
+
+		if conf.Net.TLS.Enable {
+			b.conn, b.connErr = tls.DialWithDialer(&dialer, "tcp", b.addr, conf.Net.TLS.Config)
+		} else {
+			b.conn, b.connErr = dialer.Dial("tcp", b.addr)
+		}
+		if b.connErr != nil {
+			Logger.Printf("Failed to connect to broker %s: %s\n", b.addr, b.connErr)
+			b.conn = nil
+			atomic.StoreInt32(&b.opened, 0)
+			return
+		}
+		b.conn = newBufConn(b.conn)
+
+		b.conf = conf
+
+		// Create or reuse the global metrics shared between brokers
+		b.incomingByteRate = metrics.GetOrRegisterMeter("incoming-byte-rate", conf.MetricRegistry)
+		b.requestRate = metrics.GetOrRegisterMeter("request-rate", conf.MetricRegistry)
+		b.requestSize = getOrRegisterHistogram("request-size", conf.MetricRegistry)
+		b.requestLatency = getOrRegisterHistogram("request-latency-in-ms", conf.MetricRegistry)
+		b.outgoingByteRate = metrics.GetOrRegisterMeter("outgoing-byte-rate", conf.MetricRegistry)
+		b.responseRate = metrics.GetOrRegisterMeter("response-rate", conf.MetricRegistry)
+		b.responseSize = getOrRegisterHistogram("response-size", conf.MetricRegistry)
+		// Do not gather metrics for seeded broker (only used during bootstrap) because they share
+		// the same id (-1) and are already exposed through the global metrics above
+		if b.id >= 0 {
+			b.brokerIncomingByteRate = getOrRegisterBrokerMeter("incoming-byte-rate", b, conf.MetricRegistry)
+			b.brokerRequestRate = getOrRegisterBrokerMeter("request-rate", b, conf.MetricRegistry)
+			b.brokerRequestSize = getOrRegisterBrokerHistogram("request-size", b, conf.MetricRegistry)
+			b.brokerRequestLatency = getOrRegisterBrokerHistogram("request-latency-in-ms", b, conf.MetricRegistry)
+			b.brokerOutgoingByteRate = getOrRegisterBrokerMeter("outgoing-byte-rate", b, conf.MetricRegistry)
+			b.brokerResponseRate = getOrRegisterBrokerMeter("response-rate", b, conf.MetricRegistry)
+			b.brokerResponseSize = getOrRegisterBrokerHistogram("response-size", b, conf.MetricRegistry)
+		}
+
+		if conf.Net.SASL.Enable {
+			b.connErr = b.sendAndReceiveSASLPlainAuth()
+			if b.connErr != nil {
+				err = b.conn.Close()
+				if err == nil {
+					Logger.Printf("Closed connection to broker %s\n", b.addr)
+				} else {
+					Logger.Printf("Error while closing connection to broker %s: %s\n", b.addr, err)
+				}
+				b.conn = nil
+				atomic.StoreInt32(&b.opened, 0)
+				return
+			}
+		}
+
+		b.done = make(chan bool)
+		b.responses = make(chan responsePromise, b.conf.Net.MaxOpenRequests-1)
+
+		if b.id >= 0 {
+			Logger.Printf("Connected to broker at %s (registered as #%d)\n", b.addr, b.id)
+		} else {
+			Logger.Printf("Connected to broker at %s (unregistered)\n", b.addr)
+		}
+		go withRecover(b.responseReceiver)
+	})
+
+	return nil
+}
+
+// Connected returns true if the broker is connected and false otherwise. If the broker is not
+// connected but it had tried to connect, the error from that connection attempt is also returned.
+func (b *Broker) Connected() (bool, error) {
+	b.lock.Lock()
+	defer b.lock.Unlock()
+
+	return b.conn != nil, b.connErr
+}
+
+func (b *Broker) Close() error {
+	b.lock.Lock()
+	defer b.lock.Unlock()
+
+	if b.conn == nil {
+		return ErrNotConnected
+	}
+
+	close(b.responses)
+	<-b.done
+
+	err := b.conn.Close()
+
+	b.conn = nil
+	b.connErr = nil
+	b.done = nil
+	b.responses = nil
+
+	if err == nil {
+		Logger.Printf("Closed connection to broker %s\n", b.addr)
+	} else {
+		Logger.Printf("Error while closing connection to broker %s: %s\n", b.addr, err)
+	}
+
+	atomic.StoreInt32(&b.opened, 0)
+
+	return err
+}
+
+// ID returns the broker ID retrieved from Kafka's metadata, or -1 if that is not known.
+func (b *Broker) ID() int32 {
+	return b.id
+}
+
+// Addr returns the broker address as either retrieved from Kafka's metadata or passed to NewBroker.
+func (b *Broker) Addr() string {
+	return b.addr
+}
+
+func (b *Broker) GetMetadata(request *MetadataRequest) (*MetadataResponse, error) {
+	response := new(MetadataResponse)
+
+	err := b.sendAndReceive(request, response)
+
+	if err != nil {
+		return nil, err
+	}
+
+	return response, nil
+}
+
+func (b *Broker) GetConsumerMetadata(request *ConsumerMetadataRequest) (*ConsumerMetadataResponse, error) {
+	response := new(ConsumerMetadataResponse)
+
+	err := b.sendAndReceive(request, response)
+
+	if err != nil {
+		return nil, err
+	}
+
+	return response, nil
+}
+
+func (b *Broker) GetAvailableOffsets(request *OffsetRequest) (*OffsetResponse, error) {
+	response := new(OffsetResponse)
+
+	err := b.sendAndReceive(request, response)
+
+	if err != nil {
+		return nil, err
+	}
+
+	return response, nil
+}
+
+func (b *Broker) Produce(request *ProduceRequest) (*ProduceResponse, error) {
+	var response *ProduceResponse
+	var err error
+
+	if request.RequiredAcks == NoResponse {
+		err = b.sendAndReceive(request, nil)
+	} else {
+		response = new(ProduceResponse)
+		err = b.sendAndReceive(request, response)
+	}
+
+	if err != nil {
+		return nil, err
+	}
+
+	return response, nil
+}
+
+func (b *Broker) Fetch(request *FetchRequest) (*FetchResponse, error) {
+	response := new(FetchResponse)
+
+	err := b.sendAndReceive(request, response)
+
+	if err != nil {
+		return nil, err
+	}
+
+	return response, nil
+}
+
+func (b *Broker) CommitOffset(request *OffsetCommitRequest) (*OffsetCommitResponse, error) {
+	response := new(OffsetCommitResponse)
+
+	err := b.sendAndReceive(request, response)
+
+	if err != nil {
+		return nil, err
+	}
+
+	return response, nil
+}
+
+func (b *Broker) FetchOffset(request *OffsetFetchRequest) (*OffsetFetchResponse, error) {
+	response := new(OffsetFetchResponse)
+
+	err := b.sendAndReceive(request, response)
+
+	if err != nil {
+		return nil, err
+	}
+
+	return response, nil
+}
+
+func (b *Broker) JoinGroup(request *JoinGroupRequest) (*JoinGroupResponse, error) {
+	response := new(JoinGroupResponse)
+
+	err := b.sendAndReceive(request, response)
+	if err != nil {
+		return nil, err
+	}
+
+	return response, nil
+}
+
+func (b *Broker) SyncGroup(request *SyncGroupRequest) (*SyncGroupResponse, error) {
+	response := new(SyncGroupResponse)
+
+	err := b.sendAndReceive(request, response)
+	if err != nil {
+		return nil, err
+	}
+
+	return response, nil
+}
+
+func (b *Broker) LeaveGroup(request *LeaveGroupRequest) (*LeaveGroupResponse, error) {
+	response := new(LeaveGroupResponse)
+
+	err := b.sendAndReceive(request, response)
+	if err != nil {
+		return nil, err
+	}
+
+	return response, nil
+}
+
+func (b *Broker) Heartbeat(request *HeartbeatRequest) (*HeartbeatResponse, error) {
+	response := new(HeartbeatResponse)
+
+	err := b.sendAndReceive(request, response)
+	if err != nil {
+		return nil, err
+	}
+
+	return response, nil
+}
+
+func (b *Broker) ListGroups(request *ListGroupsRequest) (*ListGroupsResponse, error) {
+	response := new(ListGroupsResponse)
+
+	err := b.sendAndReceive(request, response)
+	if err != nil {
+		return nil, err
+	}
+
+	return response, nil
+}
+
+func (b *Broker) DescribeGroups(request *DescribeGroupsRequest) (*DescribeGroupsResponse, error) {
+	response := new(DescribeGroupsResponse)
+
+	err := b.sendAndReceive(request, response)
+	if err != nil {
+		return nil, err
+	}
+
+	return response, nil
+}
+
+func (b *Broker) send(rb protocolBody, promiseResponse bool) (*responsePromise, error) {
+	b.lock.Lock()
+	defer b.lock.Unlock()
+
+	if b.conn == nil {
+		if b.connErr != nil {
+			return nil, b.connErr
+		}
+		return nil, ErrNotConnected
+	}
+
+	if !b.conf.Version.IsAtLeast(rb.requiredVersion()) {
+		return nil, ErrUnsupportedVersion
+	}
+
+	req := &request{correlationID: b.correlationID, clientID: b.conf.ClientID, body: rb}
+	buf, err := encode(req, b.conf.MetricRegistry)
+	if err != nil {
+		return nil, err
+	}
+
+	err = b.conn.SetWriteDeadline(time.Now().Add(b.conf.Net.WriteTimeout))
+	if err != nil {
+		return nil, err
+	}
+
+	requestTime := time.Now()
+	bytes, err := b.conn.Write(buf)
+	b.updateOutgoingCommunicationMetrics(bytes)
+	if err != nil {
+		return nil, err
+	}
+	b.correlationID++
+
+	if !promiseResponse {
+		// Record request latency without the response
+		b.updateRequestLatencyMetrics(time.Since(requestTime))
+		return nil, nil
+	}
+
+	promise := responsePromise{requestTime, req.correlationID, make(chan []byte), make(chan error)}
+	b.responses <- promise
+
+	return &promise, nil
+}
+
+func (b *Broker) sendAndReceive(req protocolBody, res versionedDecoder) error {
+	promise, err := b.send(req, res != nil)
+
+	if err != nil {
+		return err
+	}
+
+	if promise == nil {
+		return nil
+	}
+
+	select {
+	case buf := <-promise.packets:
+		return versionedDecode(buf, res, req.version())
+	case err = <-promise.errors:
+		return err
+	}
+}
+
+func (b *Broker) decode(pd packetDecoder) (err error) {
+	b.id, err = pd.getInt32()
+	if err != nil {
+		return err
+	}
+
+	host, err := pd.getString()
+	if err != nil {
+		return err
+	}
+
+	port, err := pd.getInt32()
+	if err != nil {
+		return err
+	}
+
+	b.addr = net.JoinHostPort(host, fmt.Sprint(port))
+	if _, _, err := net.SplitHostPort(b.addr); err != nil {
+		return err
+	}
+
+	return nil
+}
+
+func (b *Broker) encode(pe packetEncoder) (err error) {
+
+	host, portstr, err := net.SplitHostPort(b.addr)
+	if err != nil {
+		return err
+	}
+	port, err := strconv.Atoi(portstr)
+	if err != nil {
+		return err
+	}
+
+	pe.putInt32(b.id)
+
+	err = pe.putString(host)
+	if err != nil {
+		return err
+	}
+
+	pe.putInt32(int32(port))
+
+	return nil
+}
+
+func (b *Broker) responseReceiver() {
+	var dead error
+	header := make([]byte, 8)
+	for response := range b.responses {
+		if dead != nil {
+			response.errors <- dead
+			continue
+		}
+
+		err := b.conn.SetReadDeadline(time.Now().Add(b.conf.Net.ReadTimeout))
+		if err != nil {
+			dead = err
+			response.errors <- err
+			continue
+		}
+
+		bytesReadHeader, err := io.ReadFull(b.conn, header)
+		requestLatency := time.Since(response.requestTime)
+		if err != nil {
+			b.updateIncomingCommunicationMetrics(bytesReadHeader, requestLatency)
+			dead = err
+			response.errors <- err
+			continue
+		}
+
+		decodedHeader := responseHeader{}
+		err = decode(header, &decodedHeader)
+		if err != nil {
+			b.updateIncomingCommunicationMetrics(bytesReadHeader, requestLatency)
+			dead = err
+			response.errors <- err
+			continue
+		}
+		if decodedHeader.correlationID != response.correlationID {
+			b.updateIncomingCommunicationMetrics(bytesReadHeader, requestLatency)
+			// TODO if decoded ID < cur ID, discard until we catch up
+			// TODO if decoded ID > cur ID, save it so when cur ID catches up we have a response
+			dead = PacketDecodingError{fmt.Sprintf("correlation ID didn't match, wanted %d, got %d", response.correlationID, decodedHeader.correlationID)}
+			response.errors <- dead
+			continue
+		}
+
+		buf := make([]byte, decodedHeader.length-4)
+		bytesReadBody, err := io.ReadFull(b.conn, buf)
+		b.updateIncomingCommunicationMetrics(bytesReadHeader+bytesReadBody, requestLatency)
+		if err != nil {
+			dead = err
+			response.errors <- err
+			continue
+		}
+
+		response.packets <- buf
+	}
+	close(b.done)
+}
+
+func (b *Broker) sendAndReceiveSASLPlainHandshake() error {
+	rb := &SaslHandshakeRequest{"PLAIN"}
+	req := &request{correlationID: b.correlationID, clientID: b.conf.ClientID, body: rb}
+	buf, err := encode(req, b.conf.MetricRegistry)
+	if err != nil {
+		return err
+	}
+
+	err = b.conn.SetWriteDeadline(time.Now().Add(b.conf.Net.WriteTimeout))
+	if err != nil {
+		return err
+	}
+
+	requestTime := time.Now()
+	bytes, err := b.conn.Write(buf)
+	b.updateOutgoingCommunicationMetrics(bytes)
+	if err != nil {
+		Logger.Printf("Failed to send SASL handshake %s: %s\n", b.addr, err.Error())
+		return err
+	}
+	b.correlationID++
+	//wait for the response
+	header := make([]byte, 8) // response header
+	_, err = io.ReadFull(b.conn, header)
+	if err != nil {
+		Logger.Printf("Failed to read SASL handshake header : %s\n", err.Error())
+		return err
+	}
+	length := binary.BigEndian.Uint32(header[:4])
+	payload := make([]byte, length-4)
+	n, err := io.ReadFull(b.conn, payload)
+	if err != nil {
+		Logger.Printf("Failed to read SASL handshake payload : %s\n", err.Error())
+		return err
+	}
+	b.updateIncomingCommunicationMetrics(n+8, time.Since(requestTime))
+	res := &SaslHandshakeResponse{}
+	err = versionedDecode(payload, res, 0)
+	if err != nil {
+		Logger.Printf("Failed to parse SASL handshake : %s\n", err.Error())
+		return err
+	}
+	if res.Err != ErrNoError {
+		Logger.Printf("Invalid SASL Mechanism : %s\n", res.Err.Error())
+		return res.Err
+	}
+	Logger.Print("Successful SASL handshake")
+	return nil
+}
+
+// Kafka 0.10.0 plans to support SASL Plain and Kerberos as per PR #812 (KIP-43)/(JIRA KAFKA-3149)
+// Some hosted kafka services such as IBM Message Hub already offer SASL/PLAIN auth with Kafka 0.9
+//
+// In SASL Plain, Kafka expects the auth header to be in the following format
+// Message format (from https://tools.ietf.org/html/rfc4616):
+//
+//   message   = [authzid] UTF8NUL authcid UTF8NUL passwd
+//   authcid   = 1*SAFE ; MUST accept up to 255 octets
+//   authzid   = 1*SAFE ; MUST accept up to 255 octets
+//   passwd    = 1*SAFE ; MUST accept up to 255 octets
+//   UTF8NUL   = %x00 ; UTF-8 encoded NUL character
+//
+//   SAFE      = UTF1 / UTF2 / UTF3 / UTF4
+//                  ;; any UTF-8 encoded Unicode character except NUL
+//
+// When credentials are valid, Kafka returns a 4 byte array of null characters.
+// When credentials are invalid, Kafka closes the connection. This does not seem to be the ideal way
+// of responding to bad credentials but thats how its being done today.
+func (b *Broker) sendAndReceiveSASLPlainAuth() error {
+	if b.conf.Net.SASL.Handshake {
+		handshakeErr := b.sendAndReceiveSASLPlainHandshake()
+		if handshakeErr != nil {
+			Logger.Printf("Error while performing SASL handshake %s\n", b.addr)
+			return handshakeErr
+		}
+	}
+	length := 1 + len(b.conf.Net.SASL.User) + 1 + len(b.conf.Net.SASL.Password)
+	authBytes := make([]byte, length+4) //4 byte length header + auth data
+	binary.BigEndian.PutUint32(authBytes, uint32(length))
+	copy(authBytes[4:], []byte("\x00"+b.conf.Net.SASL.User+"\x00"+b.conf.Net.SASL.Password))
+
+	err := b.conn.SetWriteDeadline(time.Now().Add(b.conf.Net.WriteTimeout))
+	if err != nil {
+		Logger.Printf("Failed to set write deadline when doing SASL auth with broker %s: %s\n", b.addr, err.Error())
+		return err
+	}
+
+	requestTime := time.Now()
+	bytesWritten, err := b.conn.Write(authBytes)
+	b.updateOutgoingCommunicationMetrics(bytesWritten)
+	if err != nil {
+		Logger.Printf("Failed to write SASL auth header to broker %s: %s\n", b.addr, err.Error())
+		return err
+	}
+
+	header := make([]byte, 4)
+	n, err := io.ReadFull(b.conn, header)
+	b.updateIncomingCommunicationMetrics(n, time.Since(requestTime))
+	// If the credentials are valid, we would get a 4 byte response filled with null characters.
+	// Otherwise, the broker closes the connection and we get an EOF
+	if err != nil {
+		Logger.Printf("Failed to read response while authenticating with SASL to broker %s: %s\n", b.addr, err.Error())
+		return err
+	}
+
+	Logger.Printf("SASL authentication successful with broker %s:%v - %v\n", b.addr, n, header)
+	return nil
+}
+
+func (b *Broker) updateIncomingCommunicationMetrics(bytes int, requestLatency time.Duration) {
+	b.updateRequestLatencyMetrics(requestLatency)
+	b.responseRate.Mark(1)
+	if b.brokerResponseRate != nil {
+		b.brokerResponseRate.Mark(1)
+	}
+	responseSize := int64(bytes)
+	b.incomingByteRate.Mark(responseSize)
+	if b.brokerIncomingByteRate != nil {
+		b.brokerIncomingByteRate.Mark(responseSize)
+	}
+	b.responseSize.Update(responseSize)
+	if b.brokerResponseSize != nil {
+		b.brokerResponseSize.Update(responseSize)
+	}
+}
+
+func (b *Broker) updateRequestLatencyMetrics(requestLatency time.Duration) {
+	requestLatencyInMs := int64(requestLatency / time.Millisecond)
+	b.requestLatency.Update(requestLatencyInMs)
+	if b.brokerRequestLatency != nil {
+		b.brokerRequestLatency.Update(requestLatencyInMs)
+	}
+}
+
+func (b *Broker) updateOutgoingCommunicationMetrics(bytes int) {
+	b.requestRate.Mark(1)
+	if b.brokerRequestRate != nil {
+		b.brokerRequestRate.Mark(1)
+	}
+	requestSize := int64(bytes)
+	b.outgoingByteRate.Mark(requestSize)
+	if b.brokerOutgoingByteRate != nil {
+		b.brokerOutgoingByteRate.Mark(requestSize)
+	}
+	b.requestSize.Update(requestSize)
+	if b.brokerRequestSize != nil {
+		b.brokerRequestSize.Update(requestSize)
+	}
+}
@@ -0,0 +1,736 @@
+package sarama
+
+import (
+	"math/rand"
+	"sort"
+	"sync"
+	"time"
+)
+
+// Client is a generic Kafka client. It manages connections to one or more Kafka brokers.
+// You MUST call Close() on a client to avoid leaks, it will not be garbage-collected
+// automatically when it passes out of scope. It is safe to share a client amongst many
+// users, however Kafka will process requests from a single client strictly in serial,
+// so it is generally more efficient to use the default one client per producer/consumer.
+type Client interface {
+	// Config returns the Config struct of the client. This struct should not be
+	// altered after it has been created.
+	Config() *Config
+
+	// Topics returns the set of available topics as retrieved from cluster metadata.
+	Topics() ([]string, error)
+
+	// Partitions returns the sorted list of all partition IDs for the given topic.
+	Partitions(topic string) ([]int32, error)
+
+	// WritablePartitions returns the sorted list of all writable partition IDs for
+	// the given topic, where "writable" means "having a valid leader accepting
+	// writes".
+	WritablePartitions(topic string) ([]int32, error)
+
+	// Leader returns the broker object that is the leader of the current
+	// topic/partition, as determined by querying the cluster metadata.
+	Leader(topic string, partitionID int32) (*Broker, error)
+
+	// Replicas returns the set of all replica IDs for the given partition.
+	Replicas(topic string, partitionID int32) ([]int32, error)
+
+	// RefreshMetadata takes a list of topics and queries the cluster to refresh the
+	// available metadata for those topics. If no topics are provided, it will refresh
+	// metadata for all topics.
+	RefreshMetadata(topics ...string) error
+
+	// GetOffset queries the cluster to get the most recent available offset at the
+	// given time on the topic/partition combination. Time should be OffsetOldest for
+	// the earliest available offset, OffsetNewest for the offset of the message that
+	// will be produced next, or a time.
+	GetOffset(topic string, partitionID int32, time int64) (int64, error)
+
+	// Coordinator returns the coordinating broker for a consumer group. It will
+	// return a locally cached value if it's available. You can call
+	// RefreshCoordinator to update the cached value. This function only works on
+	// Kafka 0.8.2 and higher.
+	Coordinator(consumerGroup string) (*Broker, error)
+
+	// RefreshCoordinator retrieves the coordinator for a consumer group and stores it
+	// in local cache. This function only works on Kafka 0.8.2 and higher.
+	RefreshCoordinator(consumerGroup string) error
+
+	// Close shuts down all broker connections managed by this client. It is required
+	// to call this function before a client object passes out of scope, as it will
+	// otherwise leak memory. You must close any Producers or Consumers using a client
+	// before you close the client.
+	Close() error
+
+	// Closed returns true if the client has already had Close called on it
+	Closed() bool
+}
+
+const (
+	// OffsetNewest stands for the log head offset, i.e. the offset that will be
+	// assigned to the next message that will be produced to the partition. You
+	// can send this to a client's GetOffset method to get this offset, or when
+	// calling ConsumePartition to start consuming new messages.
+	OffsetNewest int64 = -1
+	// OffsetOldest stands for the oldest offset available on the broker for a
+	// partition. You can send this to a client's GetOffset method to get this
+	// offset, or when calling ConsumePartition to start consuming from the
+	// oldest offset that is still available on the broker.
+	OffsetOldest int64 = -2
+)
+
+type client struct {
+	conf           *Config
+	closer, closed chan none // for shutting down background metadata updater
+
+	// the broker addresses given to us through the constructor are not guaranteed to be returned in
+	// the cluster metadata (I *think* it only returns brokers who are currently leading partitions?)
+	// so we store them separately
+	seedBrokers []*Broker
+	deadSeeds   []*Broker
+
+	brokers      map[int32]*Broker                       // maps broker ids to brokers
+	metadata     map[string]map[int32]*PartitionMetadata // maps topics to partition ids to metadata
+	coordinators map[string]int32                        // Maps consumer group names to coordinating broker IDs
+
+	// If the number of partitions is large, we can get some churn calling cachedPartitions,
+	// so the result is cached.  It is important to update this value whenever metadata is changed
+	cachedPartitionsResults map[string][maxPartitionIndex][]int32
+
+	lock sync.RWMutex // protects access to the maps that hold cluster state.
+}
+
+// NewClient creates a new Client. It connects to one of the given broker addresses
+// and uses that broker to automatically fetch metadata on the rest of the kafka cluster. If metadata cannot
+// be retrieved from any of the given broker addresses, the client is not created.
+func NewClient(addrs []string, conf *Config) (Client, error) {
+	Logger.Println("Initializing new client")
+
+	if conf == nil {
+		conf = NewConfig()
+	}
+
+	if err := conf.Validate(); err != nil {
+		return nil, err
+	}
+
+	if len(addrs) < 1 {
+		return nil, ConfigurationError("You must provide at least one broker address")
+	}
+
+	client := &client{
+		conf:                    conf,
+		closer:                  make(chan none),
+		closed:                  make(chan none),
+		brokers:                 make(map[int32]*Broker),
+		metadata:                make(map[string]map[int32]*PartitionMetadata),
+		cachedPartitionsResults: make(map[string][maxPartitionIndex][]int32),
+		coordinators:            make(map[string]int32),
+	}
+
+	random := rand.New(rand.NewSource(time.Now().UnixNano()))
+	for _, index := range random.Perm(len(addrs)) {
+		client.seedBrokers = append(client.seedBrokers, NewBroker(addrs[index]))
+	}
+
+	// do an initial fetch of all cluster metadata by specifing an empty list of topics
+	err := client.RefreshMetadata()
+	switch err {
+	case nil:
+		break
+	case ErrLeaderNotAvailable, ErrReplicaNotAvailable, ErrTopicAuthorizationFailed, ErrClusterAuthorizationFailed:
+		// indicates that maybe part of the cluster is down, but is not fatal to creating the client
+		Logger.Println(err)
+	default:
+		close(client.closed) // we haven't started the background updater yet, so we have to do this manually
+		_ = client.Close()
+		return nil, err
+	}
+	go withRecover(client.backgroundMetadataUpdater)
+
+	Logger.Println("Successfully initialized new client")
+
+	return client, nil
+}
+
+func (client *client) Config() *Config {
+	return client.conf
+}
+
+func (client *client) Close() error {
+	if client.Closed() {
+		// Chances are this is being called from a defer() and the error will go unobserved
+		// so we go ahead and log the event in this case.
+		Logger.Printf("Close() called on already closed client")
+		return ErrClosedClient
+	}
+
+	// shutdown and wait for the background thread before we take the lock, to avoid races
+	close(client.closer)
+	<-client.closed
+
+	client.lock.Lock()
+	defer client.lock.Unlock()
+	Logger.Println("Closing Client")
+
+	for _, broker := range client.brokers {
+		safeAsyncClose(broker)
+	}
+
+	for _, broker := range client.seedBrokers {
+		safeAsyncClose(broker)
+	}
+
+	client.brokers = nil
+	client.metadata = nil
+
+	return nil
+}
+
+func (client *client) Closed() bool {
+	return client.brokers == nil
+}
+
+func (client *client) Topics() ([]string, error) {
+	if client.Closed() {
+		return nil, ErrClosedClient
+	}
+
+	client.lock.RLock()
+	defer client.lock.RUnlock()
+
+	ret := make([]string, 0, len(client.metadata))
+	for topic := range client.metadata {
+		ret = append(ret, topic)
+	}
+
+	return ret, nil
+}
+
+func (client *client) Partitions(topic string) ([]int32, error) {
+	if client.Closed() {
+		return nil, ErrClosedClient
+	}
+
+	partitions := client.cachedPartitions(topic, allPartitions)
+
+	if len(partitions) == 0 {
+		err := client.RefreshMetadata(topic)
+		if err != nil {
+			return nil, err
+		}
+		partitions = client.cachedPartitions(topic, allPartitions)
+	}
+
+	if partitions == nil {
+		return nil, ErrUnknownTopicOrPartition
+	}
+
+	return partitions, nil
+}
+
+func (client *client) WritablePartitions(topic string) ([]int32, error) {
+	if client.Closed() {
+		return nil, ErrClosedClient
+	}
+
+	partitions := client.cachedPartitions(topic, writablePartitions)
+
+	// len==0 catches when it's nil (no such topic) and the odd case when every single
+	// partition is undergoing leader election simultaneously. Callers have to be able to handle
+	// this function returning an empty slice (which is a valid return value) but catching it
+	// here the first time (note we *don't* catch it below where we return ErrUnknownTopicOrPartition) triggers
+	// a metadata refresh as a nicety so callers can just try again and don't have to manually
+	// trigger a refresh (otherwise they'd just keep getting a stale cached copy).
+	if len(partitions) == 0 {
+		err := client.RefreshMetadata(topic)
+		if err != nil {
+			return nil, err
+		}
+		partitions = client.cachedPartitions(topic, writablePartitions)
+	}
+
+	if partitions == nil {
+		return nil, ErrUnknownTopicOrPartition
+	}
+
+	return partitions, nil
+}
+
+func (client *client) Replicas(topic string, partitionID int32) ([]int32, error) {
+	if client.Closed() {
+		return nil, ErrClosedClient
+	}
+
+	metadata := client.cachedMetadata(topic, partitionID)
+
+	if metadata == nil {
+		err := client.RefreshMetadata(topic)
+		if err != nil {
+			return nil, err
+		}
+		metadata = client.cachedMetadata(topic, partitionID)
+	}
+
+	if metadata == nil {
+		return nil, ErrUnknownTopicOrPartition
+	}
+
+	if metadata.Err == ErrReplicaNotAvailable {
+		return nil, metadata.Err
+	}
+	return dupeAndSort(metadata.Replicas), nil
+}
+
+func (client *client) Leader(topic string, partitionID int32) (*Broker, error) {
+	if client.Closed() {
+		return nil, ErrClosedClient
+	}
+
+	leader, err := client.cachedLeader(topic, partitionID)
+
+	if leader == nil {
+		err = client.RefreshMetadata(topic)
+		if err != nil {
+			return nil, err
+		}
+		leader, err = client.cachedLeader(topic, partitionID)
+	}
+
+	return leader, err
+}
+
+func (client *client) RefreshMetadata(topics ...string) error {
+	if client.Closed() {
+		return ErrClosedClient
+	}
+
+	// Prior to 0.8.2, Kafka will throw exceptions on an empty topic and not return a proper
+	// error. This handles the case by returning an error instead of sending it
+	// off to Kafka. See: https://github.com/Shopify/sarama/pull/38#issuecomment-26362310
+	for _, topic := range topics {
+		if len(topic) == 0 {
+			return ErrInvalidTopic // this is the error that 0.8.2 and later correctly return
+		}
+	}
+
+	return client.tryRefreshMetadata(topics, client.conf.Metadata.Retry.Max)
+}
+
+func (client *client) GetOffset(topic string, partitionID int32, time int64) (int64, error) {
+	if client.Closed() {
+		return -1, ErrClosedClient
+	}
+
+	offset, err := client.getOffset(topic, partitionID, time)
+
+	if err != nil {
+		if err := client.RefreshMetadata(topic); err != nil {
+			return -1, err
+		}
+		return client.getOffset(topic, partitionID, time)
+	}
+
+	return offset, err
+}
+
+func (client *client) Coordinator(consumerGroup string) (*Broker, error) {
+	if client.Closed() {
+		return nil, ErrClosedClient
+	}
+
+	coordinator := client.cachedCoordinator(consumerGroup)
+
+	if coordinator == nil {
+		if err := client.RefreshCoordinator(consumerGroup); err != nil {
+			return nil, err
+		}
+		coordinator = client.cachedCoordinator(consumerGroup)
+	}
+
+	if coordinator == nil {
+		return nil, ErrConsumerCoordinatorNotAvailable
+	}
+
+	_ = coordinator.Open(client.conf)
+	return coordinator, nil
+}
+
+func (client *client) RefreshCoordinator(consumerGroup string) error {
+	if client.Closed() {
+		return ErrClosedClient
+	}
+
+	response, err := client.getConsumerMetadata(consumerGroup, client.conf.Metadata.Retry.Max)
+	if err != nil {
+		return err
+	}
+
+	client.lock.Lock()
+	defer client.lock.Unlock()
+	client.registerBroker(response.Coordinator)
+	client.coordinators[consumerGroup] = response.Coordinator.ID()
+	return nil
+}
+
+// private broker management helpers
+
+// registerBroker makes sure a broker received by a Metadata or Coordinator request is registered
+// in the brokers map. It returns the broker that is registered, which may be the provided broker,
+// or a previously registered Broker instance. You must hold the write lock before calling this function.
+func (client *client) registerBroker(broker *Broker) {
+	if client.brokers[broker.ID()] == nil {
+		client.brokers[broker.ID()] = broker
+		Logger.Printf("client/brokers registered new broker #%d at %s", broker.ID(), broker.Addr())
+	} else if broker.Addr() != client.brokers[broker.ID()].Addr() {
+		safeAsyncClose(client.brokers[broker.ID()])
+		client.brokers[broker.ID()] = broker
+		Logger.Printf("client/brokers replaced registered broker #%d with %s", broker.ID(), broker.Addr())
+	}
+}
+
+// deregisterBroker removes a broker from the seedsBroker list, and if it's
+// not the seedbroker, removes it from brokers map completely.
+func (client *client) deregisterBroker(broker *Broker) {
+	client.lock.Lock()
+	defer client.lock.Unlock()
+
+	if len(client.seedBrokers) > 0 && broker == client.seedBrokers[0] {
+		client.deadSeeds = append(client.deadSeeds, broker)
+		client.seedBrokers = client.seedBrokers[1:]
+	} else {
+		// we do this so that our loop in `tryRefreshMetadata` doesn't go on forever,
+		// but we really shouldn't have to; once that loop is made better this case can be
+		// removed, and the function generally can be renamed from `deregisterBroker` to
+		// `nextSeedBroker` or something
+		Logger.Printf("client/brokers deregistered broker #%d at %s", broker.ID(), broker.Addr())
+		delete(client.brokers, broker.ID())
+	}
+}
+
+func (client *client) resurrectDeadBrokers() {
+	client.lock.Lock()
+	defer client.lock.Unlock()
+
+	Logger.Printf("client/brokers resurrecting %d dead seed brokers", len(client.deadSeeds))
+	client.seedBrokers = append(client.seedBrokers, client.deadSeeds...)
+	client.deadSeeds = nil
+}
+
+func (client *client) any() *Broker {
+	client.lock.RLock()
+	defer client.lock.RUnlock()
+
+	if len(client.seedBrokers) > 0 {
+		_ = client.seedBrokers[0].Open(client.conf)
+		return client.seedBrokers[0]
+	}
+
+	// not guaranteed to be random *or* deterministic
+	for _, broker := range client.brokers {
+		_ = broker.Open(client.conf)
+		return broker
+	}
+
+	return nil
+}
+
+// private caching/lazy metadata helpers
+
+type partitionType int
+
+const (
+	allPartitions partitionType = iota
+	writablePartitions
+	// If you add any more types, update the partition cache in update()
+
+	// Ensure this is the last partition type value
+	maxPartitionIndex
+)
+
+func (client *client) cachedMetadata(topic string, partitionID int32) *PartitionMetadata {
+	client.lock.RLock()
+	defer client.lock.RUnlock()
+
+	partitions := client.metadata[topic]
+	if partitions != nil {
+		return partitions[partitionID]
+	}
+
+	return nil
+}
+
+func (client *client) cachedPartitions(topic string, partitionSet partitionType) []int32 {
+	client.lock.RLock()
+	defer client.lock.RUnlock()
+
+	partitions, exists := client.cachedPartitionsResults[topic]
+
+	if !exists {
+		return nil
+	}
+	return partitions[partitionSet]
+}
+
+func (client *client) setPartitionCache(topic string, partitionSet partitionType) []int32 {
+	partitions := client.metadata[topic]
+
+	if partitions == nil {
+		return nil
+	}
+
+	ret := make([]int32, 0, len(partitions))
+	for _, partition := range partitions {
+		if partitionSet == writablePartitions && partition.Err == ErrLeaderNotAvailable {
+			continue
+		}
+		ret = append(ret, partition.ID)
+	}
+
+	sort.Sort(int32Slice(ret))
+	return ret
+}
+
+func (client *client) cachedLeader(topic string, partitionID int32) (*Broker, error) {
+	client.lock.RLock()
+	defer client.lock.RUnlock()
+
+	partitions := client.metadata[topic]
+	if partitions != nil {
+		metadata, ok := partitions[partitionID]
+		if ok {
+			if metadata.Err == ErrLeaderNotAvailable {
+				return nil, ErrLeaderNotAvailable
+			}
+			b := client.brokers[metadata.Leader]
+			if b == nil {
+				return nil, ErrLeaderNotAvailable
+			}
+			_ = b.Open(client.conf)
+			return b, nil
+		}
+	}
+
+	return nil, ErrUnknownTopicOrPartition
+}
+
+func (client *client) getOffset(topic string, partitionID int32, time int64) (int64, error) {
+	broker, err := client.Leader(topic, partitionID)
+	if err != nil {
+		return -1, err
+	}
+
+	request := &OffsetRequest{}
+	if client.conf.Version.IsAtLeast(V0_10_1_0) {
+		request.Version = 1
+	}
+	request.AddBlock(topic, partitionID, time, 1)
+
+	response, err := broker.GetAvailableOffsets(request)
+	if err != nil {
+		_ = broker.Close()
+		return -1, err
+	}
+
+	block := response.GetBlock(topic, partitionID)
+	if block == nil {
+		_ = broker.Close()
+		return -1, ErrIncompleteResponse
+	}
+	if block.Err != ErrNoError {
+		return -1, block.Err
+	}
+	if len(block.Offsets) != 1 {
+		return -1, ErrOffsetOutOfRange
+	}
+
+	return block.Offsets[0], nil
+}
+
+// core metadata update logic
+
+func (client *client) backgroundMetadataUpdater() {
+	defer close(client.closed)
+
+	if client.conf.Metadata.RefreshFrequency == time.Duration(0) {
+		return
+	}
+
+	ticker := time.NewTicker(client.conf.Metadata.RefreshFrequency)
+	defer ticker.Stop()
+
+	for {
+		select {
+		case <-ticker.C:
+			if err := client.RefreshMetadata(); err != nil {
+				Logger.Println("Client background metadata update:", err)
+			}
+		case <-client.closer:
+			return
+		}
+	}
+}
+
+func (client *client) tryRefreshMetadata(topics []string, attemptsRemaining int) error {
+	retry := func(err error) error {
+		if attemptsRemaining > 0 {
+			Logger.Printf("client/metadata retrying after %dms... (%d attempts remaining)\n", client.conf.Metadata.Retry.Backoff/time.Millisecond, attemptsRemaining)
+			time.Sleep(client.conf.Metadata.Retry.Backoff)
+			return client.tryRefreshMetadata(topics, attemptsRemaining-1)
+		}
+		return err
+	}
+
+	for broker := client.any(); broker != nil; broker = client.any() {
+		if len(topics) > 0 {
+			Logger.Printf("client/metadata fetching metadata for %v from broker %s\n", topics, broker.addr)
+		} else {
+			Logger.Printf("client/metadata fetching metadata for all topics from broker %s\n", broker.addr)
+		}
+		response, err := broker.GetMetadata(&MetadataRequest{Topics: topics})
+
+		switch err.(type) {
+		case nil:
+			// valid response, use it
+			if shouldRetry, err := client.updateMetadata(response); shouldRetry {
+				Logger.Println("client/metadata found some partitions to be leaderless")
+				return retry(err) // note: err can be nil
+			} else {
+				return err
+			}
+
+		case PacketEncodingError:
+			// didn't even send, return the error
+			return err
+		default:
+			// some other error, remove that broker and try again
+			Logger.Println("client/metadata got error from broker while fetching metadata:", err)
+			_ = broker.Close()
+			client.deregisterBroker(broker)
+		}
+	}
+
+	Logger.Println("client/metadata no available broker to send metadata request to")
+	client.resurrectDeadBrokers()
+	return retry(ErrOutOfBrokers)
+}
+
+// if no fatal error, returns a list of topics that need retrying due to ErrLeaderNotAvailable
+func (client *client) updateMetadata(data *MetadataResponse) (retry bool, err error) {
+	client.lock.Lock()
+	defer client.lock.Unlock()
+
+	// For all the brokers we received:
+	// - if it is a new ID, save it
+	// - if it is an existing ID, but the address we have is stale, discard the old one and save it
+	// - otherwise ignore it, replacing our existing one would just bounce the connection
+	for _, broker := range data.Brokers {
+		client.registerBroker(broker)
+	}
+
+	for _, topic := range data.Topics {
+		delete(client.metadata, topic.Name)
+		delete(client.cachedPartitionsResults, topic.Name)
+
+		switch topic.Err {
+		case ErrNoError:
+			break
+		case ErrInvalidTopic, ErrTopicAuthorizationFailed: // don't retry, don't store partial results
+			err = topic.Err
+			continue
+		case ErrUnknownTopicOrPartition: // retry, do not store partial partition results
+			err = topic.Err
+			retry = true
+			continue
+		case ErrLeaderNotAvailable: // retry, but store partial partition results
+			retry = true
+			break
+		default: // don't retry, don't store partial results
+			Logger.Printf("Unexpected topic-level metadata error: %s", topic.Err)
+			err = topic.Err
+			continue
+		}
+
+		client.metadata[topic.Name] = make(map[int32]*PartitionMetadata, len(topic.Partitions))
+		for _, partition := range topic.Partitions {
+			client.metadata[topic.Name][partition.ID] = partition
+			if partition.Err == ErrLeaderNotAvailable {
+				retry = true
+			}
+		}
+
+		var partitionCache [maxPartitionIndex][]int32
+		partitionCache[allPartitions] = client.setPartitionCache(topic.Name, allPartitions)
+		partitionCache[writablePartitions] = client.setPartitionCache(topic.Name, writablePartitions)
+		client.cachedPartitionsResults[topic.Name] = partitionCache
+	}
+
+	return
+}
+
+func (client *client) cachedCoordinator(consumerGroup string) *Broker {
+	client.lock.RLock()
+	defer client.lock.RUnlock()
+	if coordinatorID, ok := client.coordinators[consumerGroup]; ok {
+		return client.brokers[coordinatorID]
+	}
+	return nil
+}
+
+func (client *client) getConsumerMetadata(consumerGroup string, attemptsRemaining int) (*ConsumerMetadataResponse, error) {
+	retry := func(err error) (*ConsumerMetadataResponse, error) {
+		if attemptsRemaining > 0 {
+			Logger.Printf("client/coordinator retrying after %dms... (%d attempts remaining)\n", client.conf.Metadata.Retry.Backoff/time.Millisecond, attemptsRemaining)
+			time.Sleep(client.conf.Metadata.Retry.Backoff)
+			return client.getConsumerMetadata(consumerGroup, attemptsRemaining-1)
+		}
+		return nil, err
+	}
+
+	for broker := client.any(); broker != nil; broker = client.any() {
+		Logger.Printf("client/coordinator requesting coordinator for consumergroup %s from %s\n", consumerGroup, broker.Addr())
+
+		request := new(ConsumerMetadataRequest)
+		request.ConsumerGroup = consumerGroup
+
+		response, err := broker.GetConsumerMetadata(request)
+
+		if err != nil {
+			Logger.Printf("client/coordinator request to broker %s failed: %s\n", broker.Addr(), err)
+
+			switch err.(type) {
+			case PacketEncodingError:
+				return nil, err
+			default:
+				_ = broker.Close()
+				client.deregisterBroker(broker)
+				continue
+			}
+		}
+
+		switch response.Err {
+		case ErrNoError:
+			Logger.Printf("client/coordinator coordinator for consumergroup %s is #%d (%s)\n", consumerGroup, response.Coordinator.ID(), response.Coordinator.Addr())
+			return response, nil
+
+		case ErrConsumerCoordinatorNotAvailable:
+			Logger.Printf("client/coordinator coordinator for consumer group %s is not available\n", consumerGroup)
+
+			// This is very ugly, but this scenario will only happen once per cluster.
+			// The __consumer_offsets topic only has to be created one time.
+			// The number of partitions not configurable, but partition 0 should always exist.
+			if _, err := client.Leader("__consumer_offsets", 0); err != nil {
+				Logger.Printf("client/coordinator the __consumer_offsets topic is not initialized completely yet. Waiting 2 seconds...\n")
+				time.Sleep(2 * time.Second)
+			}
+
+			return retry(ErrConsumerCoordinatorNotAvailable)
+		default:
+			return nil, response.Err
+		}
+	}
+
+	Logger.Println("client/coordinator no available broker to send consumer metadata request to")
+	client.resurrectDeadBrokers()
+	return retry(ErrOutOfBrokers)
+}
@@ -0,0 +1,417 @@
+package sarama
+
+import (
+	"crypto/tls"
+	"regexp"
+	"time"
+
+	"github.com/rcrowley/go-metrics"
+)
+
+const defaultClientID = "sarama"
+
+var validID = regexp.MustCompile(`\A[A-Za-z0-9._-]+\z`)
+
+// Config is used to pass multiple configuration options to Sarama's constructors.
+type Config struct {
+	// Net is the namespace for network-level properties used by the Broker, and
+	// shared by the Client/Producer/Consumer.
+	Net struct {
+		// How many outstanding requests a connection is allowed to have before
+		// sending on it blocks (default 5).
+		MaxOpenRequests int
+
+		// All three of the below configurations are similar to the
+		// `socket.timeout.ms` setting in JVM kafka. All of them default
+		// to 30 seconds.
+		DialTimeout  time.Duration // How long to wait for the initial connection.
+		ReadTimeout  time.Duration // How long to wait for a response.
+		WriteTimeout time.Duration // How long to wait for a transmit.
+
+		TLS struct {
+			// Whether or not to use TLS when connecting to the broker
+			// (defaults to false).
+			Enable bool
+			// The TLS configuration to use for secure connections if
+			// enabled (defaults to nil).
+			Config *tls.Config
+		}
+
+		// SASL based authentication with broker. While there are multiple SASL authentication methods
+		// the current implementation is limited to plaintext (SASL/PLAIN) authentication
+		SASL struct {
+			// Whether or not to use SASL authentication when connecting to the broker
+			// (defaults to false).
+			Enable bool
+			// Whether or not to send the Kafka SASL handshake first if enabled
+			// (defaults to true). You should only set this to false if you're using
+			// a non-Kafka SASL proxy.
+			Handshake bool
+			//username and password for SASL/PLAIN authentication
+			User     string
+			Password string
+		}
+
+		// KeepAlive specifies the keep-alive period for an active network connection.
+		// If zero, keep-alives are disabled. (default is 0: disabled).
+		KeepAlive time.Duration
+	}
+
+	// Metadata is the namespace for metadata management properties used by the
+	// Client, and shared by the Producer/Consumer.
+	Metadata struct {
+		Retry struct {
+			// The total number of times to retry a metadata request when the
+			// cluster is in the middle of a leader election (default 3).
+			Max int
+			// How long to wait for leader election to occur before retrying
+			// (default 250ms). Similar to the JVM's `retry.backoff.ms`.
+			Backoff time.Duration
+		}
+		// How frequently to refresh the cluster metadata in the background.
+		// Defaults to 10 minutes. Set to 0 to disable. Similar to
+		// `topic.metadata.refresh.interval.ms` in the JVM version.
+		RefreshFrequency time.Duration
+	}
+
+	// Producer is the namespace for configuration related to producing messages,
+	// used by the Producer.
+	Producer struct {
+		// The maximum permitted size of a message (defaults to 1000000). Should be
+		// set equal to or smaller than the broker's `message.max.bytes`.
+		MaxMessageBytes int
+		// The level of acknowledgement reliability needed from the broker (defaults
+		// to WaitForLocal). Equivalent to the `request.required.acks` setting of the
+		// JVM producer.
+		RequiredAcks RequiredAcks
+		// The maximum duration the broker will wait the receipt of the number of
+		// RequiredAcks (defaults to 10 seconds). This is only relevant when
+		// RequiredAcks is set to WaitForAll or a number > 1. Only supports
+		// millisecond resolution, nanoseconds will be truncated. Equivalent to
+		// the JVM producer's `request.timeout.ms` setting.
+		Timeout time.Duration
+		// The type of compression to use on messages (defaults to no compression).
+		// Similar to `compression.codec` setting of the JVM producer.
+		Compression CompressionCodec
+		// Generates partitioners for choosing the partition to send messages to
+		// (defaults to hashing the message key). Similar to the `partitioner.class`
+		// setting for the JVM producer.
+		Partitioner PartitionerConstructor
+
+		// Return specifies what channels will be populated. If they are set to true,
+		// you must read from the respective channels to prevent deadlock.
+		Return struct {
+			// If enabled, successfully delivered messages will be returned on the
+			// Successes channel (default disabled).
+			Successes bool
+
+			// If enabled, messages that failed to deliver will be returned on the
+			// Errors channel, including error (default enabled).
+			Errors bool
+		}
+
+		// The following config options control how often messages are batched up and
+		// sent to the broker. By default, messages are sent as fast as possible, and
+		// all messages received while the current batch is in-flight are placed
+		// into the subsequent batch.
+		Flush struct {
+			// The best-effort number of bytes needed to trigger a flush. Use the
+			// global sarama.MaxRequestSize to set a hard upper limit.
+			Bytes int
+			// The best-effort number of messages needed to trigger a flush. Use
+			// `MaxMessages` to set a hard upper limit.
+			Messages int
+			// The best-effort frequency of flushes. Equivalent to
+			// `queue.buffering.max.ms` setting of JVM producer.
+			Frequency time.Duration
+			// The maximum number of messages the producer will send in a single
+			// broker request. Defaults to 0 for unlimited. Similar to
+			// `queue.buffering.max.messages` in the JVM producer.
+			MaxMessages int
+		}
+
+		Retry struct {
+			// The total number of times to retry sending a message (default 3).
+			// Similar to the `message.send.max.retries` setting of the JVM producer.
+			Max int
+			// How long to wait for the cluster to settle between retries
+			// (default 100ms). Similar to the `retry.backoff.ms` setting of the
+			// JVM producer.
+			Backoff time.Duration
+		}
+	}
+
+	// Consumer is the namespace for configuration related to consuming messages,
+	// used by the Consumer.
+	//
+	// Note that Sarama's Consumer type does not currently support automatic
+	// consumer-group rebalancing and offset tracking.  For Zookeeper-based
+	// tracking (Kafka 0.8.2 and earlier), the https://github.com/wvanbergen/kafka
+	// library builds on Sarama to add this support. For Kafka-based tracking
+	// (Kafka 0.9 and later), the https://github.com/bsm/sarama-cluster library
+	// builds on Sarama to add this support.
+	Consumer struct {
+		Retry struct {
+			// How long to wait after a failing to read from a partition before
+			// trying again (default 2s).
+			Backoff time.Duration
+		}
+
+		// Fetch is the namespace for controlling how many bytes are retrieved by any
+		// given request.
+		Fetch struct {
+			// The minimum number of message bytes to fetch in a request - the broker
+			// will wait until at least this many are available. The default is 1,
+			// as 0 causes the consumer to spin when no messages are available.
+			// Equivalent to the JVM's `fetch.min.bytes`.
+			Min int32
+			// The default number of message bytes to fetch from the broker in each
+			// request (default 32768). This should be larger than the majority of
+			// your messages, or else the consumer will spend a lot of time
+			// negotiating sizes and not actually consuming. Similar to the JVM's
+			// `fetch.message.max.bytes`.
+			Default int32
+			// The maximum number of message bytes to fetch from the broker in a
+			// single request. Messages larger than this will return
+			// ErrMessageTooLarge and will not be consumable, so you must be sure
+			// this is at least as large as your largest message. Defaults to 0
+			// (no limit). Similar to the JVM's `fetch.message.max.bytes`. The
+			// global `sarama.MaxResponseSize` still applies.
+			Max int32
+		}
+		// The maximum amount of time the broker will wait for Consumer.Fetch.Min
+		// bytes to become available before it returns fewer than that anyways. The
+		// default is 250ms, since 0 causes the consumer to spin when no events are
+		// available. 100-500ms is a reasonable range for most cases. Kafka only
+		// supports precision up to milliseconds; nanoseconds will be truncated.
+		// Equivalent to the JVM's `fetch.wait.max.ms`.
+		MaxWaitTime time.Duration
+
+		// The maximum amount of time the consumer expects a message takes to process
+		// for the user. If writing to the Messages channel takes longer than this,
+		// that partition will stop fetching more messages until it can proceed again.
+		// Note that, since the Messages channel is buffered, the actual grace time is
+		// (MaxProcessingTime * ChanneBufferSize). Defaults to 100ms.
+		MaxProcessingTime time.Duration
+
+		// Return specifies what channels will be populated. If they are set to true,
+		// you must read from them to prevent deadlock.
+		Return struct {
+			// If enabled, any errors that occurred while consuming are returned on
+			// the Errors channel (default disabled).
+			Errors bool
+		}
+
+		// Offsets specifies configuration for how and when to commit consumed
+		// offsets. This currently requires the manual use of an OffsetManager
+		// but will eventually be automated.
+		Offsets struct {
+			// How frequently to commit updated offsets. Defaults to 1s.
+			CommitInterval time.Duration
+
+			// The initial offset to use if no offset was previously committed.
+			// Should be OffsetNewest or OffsetOldest. Defaults to OffsetNewest.
+			Initial int64
+
+			// The retention duration for committed offsets. If zero, disabled
+			// (in which case the `offsets.retention.minutes` option on the
+			// broker will be used).  Kafka only supports precision up to
+			// milliseconds; nanoseconds will be truncated. Requires Kafka
+			// broker version 0.9.0 or later.
+			// (default is 0: disabled).
+			Retention time.Duration
+		}
+	}
+
+	// A user-provided string sent with every request to the brokers for logging,
+	// debugging, and auditing purposes. Defaults to "sarama", but you should
+	// probably set it to something specific to your application.
+	ClientID string
+	// The number of events to buffer in internal and external channels. This
+	// permits the producer and consumer to continue processing some messages
+	// in the background while user code is working, greatly improving throughput.
+	// Defaults to 256.
+	ChannelBufferSize int
+	// The version of Kafka that Sarama will assume it is running against.
+	// Defaults to the oldest supported stable version. Since Kafka provides
+	// backwards-compatibility, setting it to a version older than you have
+	// will not break anything, although it may prevent you from using the
+	// latest features. Setting it to a version greater than you are actually
+	// running may lead to random breakage.
+	Version KafkaVersion
+	// The registry to define metrics into.
+	// Defaults to a local registry.
+	// If you want to disable metrics gathering, set "metrics.UseNilMetrics" to "true"
+	// prior to starting Sarama.
+	// See Examples on how to use the metrics registry
+	MetricRegistry metrics.Registry
+}
+
+// NewConfig returns a new configuration instance with sane defaults.
+func NewConfig() *Config {
+	c := &Config{}
+
+	c.Net.MaxOpenRequests = 5
+	c.Net.DialTimeout = 30 * time.Second
+	c.Net.ReadTimeout = 30 * time.Second
+	c.Net.WriteTimeout = 30 * time.Second
+	c.Net.SASL.Handshake = true
+
+	c.Metadata.Retry.Max = 3
+	c.Metadata.Retry.Backoff = 250 * time.Millisecond
+	c.Metadata.RefreshFrequency = 10 * time.Minute
+
+	c.Producer.MaxMessageBytes = 1000000
+	c.Producer.RequiredAcks = WaitForLocal
+	c.Producer.Timeout = 10 * time.Second
+	c.Producer.Partitioner = NewHashPartitioner
+	c.Producer.Retry.Max = 3
+	c.Producer.Retry.Backoff = 100 * time.Millisecond
+	c.Producer.Return.Errors = true
+
+	c.Consumer.Fetch.Min = 1
+	c.Consumer.Fetch.Default = 32768
+	c.Consumer.Retry.Backoff = 2 * time.Second
+	c.Consumer.MaxWaitTime = 250 * time.Millisecond
+	c.Consumer.MaxProcessingTime = 100 * time.Millisecond
+	c.Consumer.Return.Errors = false
+	c.Consumer.Offsets.CommitInterval = 1 * time.Second
+	c.Consumer.Offsets.Initial = OffsetNewest
+
+	c.ClientID = defaultClientID
+	c.ChannelBufferSize = 256
+	c.Version = minVersion
+	c.MetricRegistry = metrics.NewRegistry()
+
+	return c
+}
+
+// Validate checks a Config instance. It will return a
+// ConfigurationError if the specified values don't make sense.
+func (c *Config) Validate() error {
+	// some configuration values should be warned on but not fail completely, do those first
+	if c.Net.TLS.Enable == false && c.Net.TLS.Config != nil {
+		Logger.Println("Net.TLS is disabled but a non-nil configuration was provided.")
+	}
+	if c.Net.SASL.Enable == false {
+		if c.Net.SASL.User != "" {
+			Logger.Println("Net.SASL is disabled but a non-empty username was provided.")
+		}
+		if c.Net.SASL.Password != "" {
+			Logger.Println("Net.SASL is disabled but a non-empty password was provided.")
+		}
+	}
+	if c.Producer.RequiredAcks > 1 {
+		Logger.Println("Producer.RequiredAcks > 1 is deprecated and will raise an exception with kafka >= 0.8.2.0.")
+	}
+	if c.Producer.MaxMessageBytes >= int(MaxRequestSize) {
+		Logger.Println("Producer.MaxMessageBytes is larger than MaxRequestSize; it will be ignored.")
+	}
+	if c.Producer.Flush.Bytes >= int(MaxRequestSize) {
+		Logger.Println("Producer.Flush.Bytes is larger than MaxRequestSize; it will be ignored.")
+	}
+	if c.Producer.Timeout%time.Millisecond != 0 {
+		Logger.Println("Producer.Timeout only supports millisecond resolution; nanoseconds will be truncated.")
+	}
+	if c.Consumer.MaxWaitTime < 100*time.Millisecond {
+		Logger.Println("Consumer.MaxWaitTime is very low, which can cause high CPU and network usage. See documentation for details.")
+	}
+	if c.Consumer.MaxWaitTime%time.Millisecond != 0 {
+		Logger.Println("Consumer.MaxWaitTime only supports millisecond precision; nanoseconds will be truncated.")
+	}
+	if c.Consumer.Offsets.Retention%time.Millisecond != 0 {
+		Logger.Println("Consumer.Offsets.Retention only supports millisecond precision; nanoseconds will be truncated.")
+	}
+	if c.ClientID == defaultClientID {
+		Logger.Println("ClientID is the default of 'sarama', you should consider setting it to something application-specific.")
+	}
+
+	// validate Net values
+	switch {
+	case c.Net.MaxOpenRequests <= 0:
+		return ConfigurationError("Net.MaxOpenRequests must be > 0")
+	case c.Net.DialTimeout <= 0:
+		return ConfigurationError("Net.DialTimeout must be > 0")
+	case c.Net.ReadTimeout <= 0:
+		return ConfigurationError("Net.ReadTimeout must be > 0")
+	case c.Net.WriteTimeout <= 0:
+		return ConfigurationError("Net.WriteTimeout must be > 0")
+	case c.Net.KeepAlive < 0:
+		return ConfigurationError("Net.KeepAlive must be >= 0")
+	case c.Net.SASL.Enable == true && c.Net.SASL.User == "":
+		return ConfigurationError("Net.SASL.User must not be empty when SASL is enabled")
+	case c.Net.SASL.Enable == true && c.Net.SASL.Password == "":
+		return ConfigurationError("Net.SASL.Password must not be empty when SASL is enabled")
+	}
+
+	// validate the Metadata values
+	switch {
+	case c.Metadata.Retry.Max < 0:
+		return ConfigurationError("Metadata.Retry.Max must be >= 0")
+	case c.Metadata.Retry.Backoff < 0:
+		return ConfigurationError("Metadata.Retry.Backoff must be >= 0")
+	case c.Metadata.RefreshFrequency < 0:
+		return ConfigurationError("Metadata.RefreshFrequency must be >= 0")
+	}
+
+	// validate the Producer values
+	switch {
+	case c.Producer.MaxMessageBytes <= 0:
+		return ConfigurationError("Producer.MaxMessageBytes must be > 0")
+	case c.Producer.RequiredAcks < -1:
+		return ConfigurationError("Producer.RequiredAcks must be >= -1")
+	case c.Producer.Timeout <= 0:
+		return ConfigurationError("Producer.Timeout must be > 0")
+	case c.Producer.Partitioner == nil:
+		return ConfigurationError("Producer.Partitioner must not be nil")
+	case c.Producer.Flush.Bytes < 0:
+		return ConfigurationError("Producer.Flush.Bytes must be >= 0")
+	case c.Producer.Flush.Messages < 0:
+		return ConfigurationError("Producer.Flush.Messages must be >= 0")
+	case c.Producer.Flush.Frequency < 0:
+		return ConfigurationError("Producer.Flush.Frequency must be >= 0")
+	case c.Producer.Flush.MaxMessages < 0:
+		return ConfigurationError("Producer.Flush.MaxMessages must be >= 0")
+	case c.Producer.Flush.MaxMessages > 0 && c.Producer.Flush.MaxMessages < c.Producer.Flush.Messages:
+		return ConfigurationError("Producer.Flush.MaxMessages must be >= Producer.Flush.Messages when set")
+	case c.Producer.Retry.Max < 0:
+		return ConfigurationError("Producer.Retry.Max must be >= 0")
+	case c.Producer.Retry.Backoff < 0:
+		return ConfigurationError("Producer.Retry.Backoff must be >= 0")
+	}
+
+	if c.Producer.Compression == CompressionLZ4 && !c.Version.IsAtLeast(V0_10_0_0) {
+		return ConfigurationError("lz4 compression requires Version >= V0_10_0_0")
+	}
+
+	// validate the Consumer values
+	switch {
+	case c.Consumer.Fetch.Min <= 0:
+		return ConfigurationError("Consumer.Fetch.Min must be > 0")
+	case c.Consumer.Fetch.Default <= 0:
+		return ConfigurationError("Consumer.Fetch.Default must be > 0")
+	case c.Consumer.Fetch.Max < 0:
+		return ConfigurationError("Consumer.Fetch.Max must be >= 0")
+	case c.Consumer.MaxWaitTime < 1*time.Millisecond:
+		return ConfigurationError("Consumer.MaxWaitTime must be >= 1ms")
+	case c.Consumer.MaxProcessingTime <= 0:
+		return ConfigurationError("Consumer.MaxProcessingTime must be > 0")
+	case c.Consumer.Retry.Backoff < 0:
+		return ConfigurationError("Consumer.Retry.Backoff must be >= 0")
+	case c.Consumer.Offsets.CommitInterval <= 0:
+		return ConfigurationError("Consumer.Offsets.CommitInterval must be > 0")
+	case c.Consumer.Offsets.Initial != OffsetOldest && c.Consumer.Offsets.Initial != OffsetNewest:
+		return ConfigurationError("Consumer.Offsets.Initial must be OffsetOldest or OffsetNewest")
+
+	}
+
+	// validate misc shared values
+	switch {
+	case c.ChannelBufferSize < 0:
+		return ConfigurationError("ChannelBufferSize must be >= 0")
+	case !validID.MatchString(c.ClientID):
+		return ConfigurationError("ClientID is invalid")
+	}
+
+	return nil
+}
@@ -0,0 +1,735 @@
+package sarama
+
+import (
+	"errors"
+	"fmt"
+	"sync"
+	"sync/atomic"
+	"time"
+)
+
+// ConsumerMessage encapsulates a Kafka message returned by the consumer.
+type ConsumerMessage struct {
+	Key, Value []byte
+	Topic      string
+	Partition  int32
+	Offset     int64
+	Timestamp  time.Time // only set if kafka is version 0.10+
+}
+
+// ConsumerError is what is provided to the user when an error occurs.
+// It wraps an error and includes the topic and partition.
+type ConsumerError struct {
+	Topic     string
+	Partition int32
+	Err       error
+}
+
+func (ce ConsumerError) Error() string {
+	return fmt.Sprintf("kafka: error while consuming %s/%d: %s", ce.Topic, ce.Partition, ce.Err)
+}
+
+// ConsumerErrors is a type that wraps a batch of errors and implements the Error interface.
+// It can be returned from the PartitionConsumer's Close methods to avoid the need to manually drain errors
+// when stopping.
+type ConsumerErrors []*ConsumerError
+
+func (ce ConsumerErrors) Error() string {
+	return fmt.Sprintf("kafka: %d errors while consuming", len(ce))
+}
+
+// Consumer manages PartitionConsumers which process Kafka messages from brokers. You MUST call Close()
+// on a consumer to avoid leaks, it will not be garbage-collected automatically when it passes out of
+// scope.
+//
+// Sarama's Consumer type does not currently support automatic consumer-group rebalancing and offset tracking.
+// For Zookeeper-based tracking (Kafka 0.8.2 and earlier), the https://github.com/wvanbergen/kafka library
+// builds on Sarama to add this support. For Kafka-based tracking (Kafka 0.9 and later), the
+// https://github.com/bsm/sarama-cluster library builds on Sarama to add this support.
+type Consumer interface {
+
+	// Topics returns the set of available topics as retrieved from the cluster
+	// metadata. This method is the same as Client.Topics(), and is provided for
+	// convenience.
+	Topics() ([]string, error)
+
+	// Partitions returns the sorted list of all partition IDs for the given topic.
+	// This method is the same as Client.Partitions(), and is provided for convenience.
+	Partitions(topic string) ([]int32, error)
+
+	// ConsumePartition creates a PartitionConsumer on the given topic/partition with
+	// the given offset. It will return an error if this Consumer is already consuming
+	// on the given topic/partition. Offset can be a literal offset, or OffsetNewest
+	// or OffsetOldest
+	ConsumePartition(topic string, partition int32, offset int64) (PartitionConsumer, error)
+
+	// HighWaterMarks returns the current high water marks for each topic and partition.
+	// Consistency between partitions is not guaranteed since high water marks are updated separately.
+	HighWaterMarks() map[string]map[int32]int64
+
+	// Close shuts down the consumer. It must be called after all child
+	// PartitionConsumers have already been closed.
+	Close() error
+}
+
+type consumer struct {
+	client    Client
+	conf      *Config
+	ownClient bool
+
+	lock            sync.Mutex
+	children        map[string]map[int32]*partitionConsumer
+	brokerConsumers map[*Broker]*brokerConsumer
+}
+
+// NewConsumer creates a new consumer using the given broker addresses and configuration.
+func NewConsumer(addrs []string, config *Config) (Consumer, error) {
+	client, err := NewClient(addrs, config)
+	if err != nil {
+		return nil, err
+	}
+
+	c, err := NewConsumerFromClient(client)
+	if err != nil {
+		return nil, err
+	}
+	c.(*consumer).ownClient = true
+	return c, nil
+}
+
+// NewConsumerFromClient creates a new consumer using the given client. It is still
+// necessary to call Close() on the underlying client when shutting down this consumer.
+func NewConsumerFromClient(client Client) (Consumer, error) {
+	// Check that we are not dealing with a closed Client before processing any other arguments
+	if client.Closed() {
+		return nil, ErrClosedClient
+	}
+
+	c := &consumer{
+		client:          client,
+		conf:            client.Config(),
+		children:        make(map[string]map[int32]*partitionConsumer),
+		brokerConsumers: make(map[*Broker]*brokerConsumer),
+	}
+
+	return c, nil
+}
+
+func (c *consumer) Close() error {
+	if c.ownClient {
+		return c.client.Close()
+	}
+	return nil
+}
+
+func (c *consumer) Topics() ([]string, error) {
+	return c.client.Topics()
+}
+
+func (c *consumer) Partitions(topic string) ([]int32, error) {
+	return c.client.Partitions(topic)
+}
+
+func (c *consumer) ConsumePartition(topic string, partition int32, offset int64) (PartitionConsumer, error) {
+	child := &partitionConsumer{
+		consumer:  c,
+		conf:      c.conf,
+		topic:     topic,
+		partition: partition,
+		messages:  make(chan *ConsumerMessage, c.conf.ChannelBufferSize),
+		errors:    make(chan *ConsumerError, c.conf.ChannelBufferSize),
+		feeder:    make(chan *FetchResponse, 1),
+		trigger:   make(chan none, 1),
+		dying:     make(chan none),
+		fetchSize: c.conf.Consumer.Fetch.Default,
+	}
+
+	if err := child.chooseStartingOffset(offset); err != nil {
+		return nil, err
+	}
+
+	var leader *Broker
+	var err error
+	if leader, err = c.client.Leader(child.topic, child.partition); err != nil {
+		return nil, err
+	}
+
+	if err := c.addChild(child); err != nil {
+		return nil, err
+	}
+
+	go withRecover(child.dispatcher)
+	go withRecover(child.responseFeeder)
+
+	child.broker = c.refBrokerConsumer(leader)
+	child.broker.input <- child
+
+	return child, nil
+}
+
+func (c *consumer) HighWaterMarks() map[string]map[int32]int64 {
+	c.lock.Lock()
+	defer c.lock.Unlock()
+
+	hwms := make(map[string]map[int32]int64)
+	for topic, p := range c.children {
+		hwm := make(map[int32]int64, len(p))
+		for partition, pc := range p {
+			hwm[partition] = pc.HighWaterMarkOffset()
+		}
+		hwms[topic] = hwm
+	}
+
+	return hwms
+}
+
+func (c *consumer) addChild(child *partitionConsumer) error {
+	c.lock.Lock()
+	defer c.lock.Unlock()
+
+	topicChildren := c.children[child.topic]
+	if topicChildren == nil {
+		topicChildren = make(map[int32]*partitionConsumer)
+		c.children[child.topic] = topicChildren
+	}
+
+	if topicChildren[child.partition] != nil {
+		return ConfigurationError("That topic/partition is already being consumed")
+	}
+
+	topicChildren[child.partition] = child
+	return nil
+}
+
+func (c *consumer) removeChild(child *partitionConsumer) {
+	c.lock.Lock()
+	defer c.lock.Unlock()
+
+	delete(c.children[child.topic], child.partition)
+}
+
+func (c *consumer) refBrokerConsumer(broker *Broker) *brokerConsumer {
+	c.lock.Lock()
+	defer c.lock.Unlock()
+
+	bc := c.brokerConsumers[broker]
+	if bc == nil {
+		bc = c.newBrokerConsumer(broker)
+		c.brokerConsumers[broker] = bc
+	}
+
+	bc.refs++
+
+	return bc
+}
+
+func (c *consumer) unrefBrokerConsumer(brokerWorker *brokerConsumer) {
+	c.lock.Lock()
+	defer c.lock.Unlock()
+
+	brokerWorker.refs--
+
+	if brokerWorker.refs == 0 {
+		close(brokerWorker.input)
+		if c.brokerConsumers[brokerWorker.broker] == brokerWorker {
+			delete(c.brokerConsumers, brokerWorker.broker)
+		}
+	}
+}
+
+func (c *consumer) abandonBrokerConsumer(brokerWorker *brokerConsumer) {
+	c.lock.Lock()
+	defer c.lock.Unlock()
+
+	delete(c.brokerConsumers, brokerWorker.broker)
+}
+
+// PartitionConsumer
+
+// PartitionConsumer processes Kafka messages from a given topic and partition. You MUST call Close()
+// or AsyncClose() on a PartitionConsumer to avoid leaks, it will not be garbage-collected automatically
+// when it passes out of scope.
+//
+// The simplest way of using a PartitionConsumer is to loop over its Messages channel using a for/range
+// loop. The PartitionConsumer will only stop itself in one case: when the offset being consumed is reported
+// as out of range by the brokers. In this case you should decide what you want to do (try a different offset,
+// notify a human, etc) and handle it appropriately. For all other error cases, it will just keep retrying.
+// By default, it logs these errors to sarama.Logger; if you want to be notified directly of all errors, set
+// your config's Consumer.Return.Errors to true and read from the Errors channel, using a select statement
+// or a separate goroutine. Check out the Consumer examples to see implementations of these different approaches.
+type PartitionConsumer interface {
+
+	// AsyncClose initiates a shutdown of the PartitionConsumer. This method will
+	// return immediately, after which you should wait until the 'messages' and
+	// 'errors' channel are drained. It is required to call this function, or
+	// Close before a consumer object passes out of scope, as it will otherwise
+	// leak memory. You must call this before calling Close on the underlying client.
+	AsyncClose()
+
+	// Close stops the PartitionConsumer from fetching messages. It is required to
+	// call this function (or AsyncClose) before a consumer object passes out of
+	// scope, as it will otherwise leak memory. You must call this before calling
+	// Close on the underlying client.
+	Close() error
+
+	// Messages returns the read channel for the messages that are returned by
+	// the broker.
+	Messages() <-chan *ConsumerMessage
+
+	// Errors returns a read channel of errors that occurred during consuming, if
+	// enabled. By default, errors are logged and not returned over this channel.
+	// If you want to implement any custom error handling, set your config's
+	// Consumer.Return.Errors setting to true, and read from this channel.
+	Errors() <-chan *ConsumerError
+
+	// HighWaterMarkOffset returns the high water mark offset of the partition,
+	// i.e. the offset that will be used for the next message that will be produced.
+	// You can use this to determine how far behind the processing is.
+	HighWaterMarkOffset() int64
+}
+
+type partitionConsumer struct {
+	consumer  *consumer
+	conf      *Config
+	topic     string
+	partition int32
+
+	broker   *brokerConsumer
+	messages chan *ConsumerMessage
+	errors   chan *ConsumerError
+	feeder   chan *FetchResponse
+
+	trigger, dying chan none
+	responseResult error
+
+	fetchSize           int32
+	offset              int64
+	highWaterMarkOffset int64
+}
+
+var errTimedOut = errors.New("timed out feeding messages to the user") // not user-facing
+
+func (child *partitionConsumer) sendError(err error) {
+	cErr := &ConsumerError{
+		Topic:     child.topic,
+		Partition: child.partition,
+		Err:       err,
+	}
+
+	if child.conf.Consumer.Return.Errors {
+		child.errors <- cErr
+	} else {
+		Logger.Println(cErr)
+	}
+}
+
+func (child *partitionConsumer) dispatcher() {
+	for _ = range child.trigger {
+		select {
+		case <-child.dying:
+			close(child.trigger)
+		case <-time.After(child.conf.Consumer.Retry.Backoff):
+			if child.broker != nil {
+				child.consumer.unrefBrokerConsumer(child.broker)
+				child.broker = nil
+			}
+
+			Logger.Printf("consumer/%s/%d finding new broker\n", child.topic, child.partition)
+			if err := child.dispatch(); err != nil {
+				child.sendError(err)
+				child.trigger <- none{}
+			}
+		}
+	}
+
+	if child.broker != nil {
+		child.consumer.unrefBrokerConsumer(child.broker)
+	}
+	child.consumer.removeChild(child)
+	close(child.feeder)
+}
+
+func (child *partitionConsumer) dispatch() error {
+	if err := child.consumer.client.RefreshMetadata(child.topic); err != nil {
+		return err
+	}
+
+	var leader *Broker
+	var err error
+	if leader, err = child.consumer.client.Leader(child.topic, child.partition); err != nil {
+		return err
+	}
+
+	child.broker = child.consumer.refBrokerConsumer(leader)
+
+	child.broker.input <- child
+
+	return nil
+}
+
+func (child *partitionConsumer) chooseStartingOffset(offset int64) error {
+	newestOffset, err := child.consumer.client.GetOffset(child.topic, child.partition, OffsetNewest)
+	if err != nil {
+		return err
+	}
+	oldestOffset, err := child.consumer.client.GetOffset(child.topic, child.partition, OffsetOldest)
+	if err != nil {
+		return err
+	}
+
+	switch {
+	case offset == OffsetNewest:
+		child.offset = newestOffset
+	case offset == OffsetOldest:
+		child.offset = oldestOffset
+	case offset >= oldestOffset && offset <= newestOffset:
+		child.offset = offset
+	default:
+		return ErrOffsetOutOfRange
+	}
+
+	return nil
+}
+
+func (child *partitionConsumer) Messages() <-chan *ConsumerMessage {
+	return child.messages
+}
+
+func (child *partitionConsumer) Errors() <-chan *ConsumerError {
+	return child.errors
+}
+
+func (child *partitionConsumer) AsyncClose() {
+	// this triggers whatever broker owns this child to abandon it and close its trigger channel, which causes
+	// the dispatcher to exit its loop, which removes it from the consumer then closes its 'messages' and
+	// 'errors' channel (alternatively, if the child is already at the dispatcher for some reason, that will
+	// also just close itself)
+	close(child.dying)
+}
+
+func (child *partitionConsumer) Close() error {
+	child.AsyncClose()
+
+	go withRecover(func() {
+		for _ = range child.messages {
+			// drain
+		}
+	})
+
+	var errors ConsumerErrors
+	for err := range child.errors {
+		errors = append(errors, err)
+	}
+
+	if len(errors) > 0 {
+		return errors
+	}
+	return nil
+}
+
+func (child *partitionConsumer) HighWaterMarkOffset() int64 {
+	return atomic.LoadInt64(&child.highWaterMarkOffset)
+}
+
+func (child *partitionConsumer) responseFeeder() {
+	var msgs []*ConsumerMessage
+	expiryTimer := time.NewTimer(child.conf.Consumer.MaxProcessingTime)
+	expireTimedOut := false
+
+feederLoop:
+	for response := range child.feeder {
+		msgs, child.responseResult = child.parseResponse(response)
+
+		for i, msg := range msgs {
+			if !expiryTimer.Stop() && !expireTimedOut {
+				// expiryTimer was expired; clear out the waiting msg
+				<-expiryTimer.C
+			}
+			expiryTimer.Reset(child.conf.Consumer.MaxProcessingTime)
+			expireTimedOut = false
+
+			select {
+			case child.messages <- msg:
+			case <-expiryTimer.C:
+				expireTimedOut = true
+				child.responseResult = errTimedOut
+				child.broker.acks.Done()
+				for _, msg = range msgs[i:] {
+					child.messages <- msg
+				}
+				child.broker.input <- child
+				continue feederLoop
+			}
+		}
+
+		child.broker.acks.Done()
+	}
+
+	close(child.messages)
+	close(child.errors)
+}
+
+func (child *partitionConsumer) parseResponse(response *FetchResponse) ([]*ConsumerMessage, error) {
+	block := response.GetBlock(child.topic, child.partition)
+	if block == nil {
+		return nil, ErrIncompleteResponse
+	}
+
+	if block.Err != ErrNoError {
+		return nil, block.Err
+	}
+
+	if len(block.MsgSet.Messages) == 0 {
+		// We got no messages. If we got a trailing one then we need to ask for more data.
+		// Otherwise we just poll again and wait for one to be produced...
+		if block.MsgSet.PartialTrailingMessage {
+			if child.conf.Consumer.Fetch.Max > 0 && child.fetchSize == child.conf.Consumer.Fetch.Max {
+				// we can't ask for more data, we've hit the configured limit
+				child.sendError(ErrMessageTooLarge)
+				child.offset++ // skip this one so we can keep processing future messages
+			} else {
+				child.fetchSize *= 2
+				if child.conf.Consumer.Fetch.Max > 0 && child.fetchSize > child.conf.Consumer.Fetch.Max {
+					child.fetchSize = child.conf.Consumer.Fetch.Max
+				}
+			}
+		}
+
+		return nil, nil
+	}
+
+	// we got messages, reset our fetch size in case it was increased for a previous request
+	child.fetchSize = child.conf.Consumer.Fetch.Default
+	atomic.StoreInt64(&child.highWaterMarkOffset, block.HighWaterMarkOffset)
+
+	incomplete := false
+	prelude := true
+	var messages []*ConsumerMessage
+	for _, msgBlock := range block.MsgSet.Messages {
+
+		for _, msg := range msgBlock.Messages() {
+			offset := msg.Offset
+			if msg.Msg.Version >= 1 {
+				baseOffset := msgBlock.Offset - msgBlock.Messages()[len(msgBlock.Messages())-1].Offset
+				offset += baseOffset
+			}
+			if prelude && offset < child.offset {
+				continue
+			}
+			prelude = false
+
+			if offset >= child.offset {
+				messages = append(messages, &ConsumerMessage{
+					Topic:     child.topic,
+					Partition: child.partition,
+					Key:       msg.Msg.Key,
+					Value:     msg.Msg.Value,
+					Offset:    offset,
+					Timestamp: msg.Msg.Timestamp,
+				})
+				child.offset = offset + 1
+			} else {
+				incomplete = true
+			}
+		}
+
+	}
+
+	if incomplete || len(messages) == 0 {
+		return nil, ErrIncompleteResponse
+	}
+	return messages, nil
+}
+
+// brokerConsumer
+
+type brokerConsumer struct {
+	consumer         *consumer
+	broker           *Broker
+	input            chan *partitionConsumer
+	newSubscriptions chan []*partitionConsumer
+	wait             chan none
+	subscriptions    map[*partitionConsumer]none
+	acks             sync.WaitGroup
+	refs             int
+}
+
+func (c *consumer) newBrokerConsumer(broker *Broker) *brokerConsumer {
+	bc := &brokerConsumer{
+		consumer:         c,
+		broker:           broker,
+		input:            make(chan *partitionConsumer),
+		newSubscriptions: make(chan []*partitionConsumer),
+		wait:             make(chan none),
+		subscriptions:    make(map[*partitionConsumer]none),
+		refs:             0,
+	}
+
+	go withRecover(bc.subscriptionManager)
+	go withRecover(bc.subscriptionConsumer)
+
+	return bc
+}
+
+func (bc *brokerConsumer) subscriptionManager() {
+	var buffer []*partitionConsumer
+
+	// The subscriptionManager constantly accepts new subscriptions on `input` (even when the main subscriptionConsumer
+	// goroutine is in the middle of a network request) and batches it up. The main worker goroutine picks
+	// up a batch of new subscriptions between every network request by reading from `newSubscriptions`, so we give
+	// it nil if no new subscriptions are available. We also write to `wait` only when new subscriptions is available,
+	// so the main goroutine can block waiting for work if it has none.
+	for {
+		if len(buffer) > 0 {
+			select {
+			case event, ok := <-bc.input:
+				if !ok {
+					goto done
+				}
+				buffer = append(buffer, event)
+			case bc.newSubscriptions <- buffer:
+				buffer = nil
+			case bc.wait <- none{}:
+			}
+		} else {
+			select {
+			case event, ok := <-bc.input:
+				if !ok {
+					goto done
+				}
+				buffer = append(buffer, event)
+			case bc.newSubscriptions <- nil:
+			}
+		}
+	}
+
+done:
+	close(bc.wait)
+	if len(buffer) > 0 {
+		bc.newSubscriptions <- buffer
+	}
+	close(bc.newSubscriptions)
+}
+
+func (bc *brokerConsumer) subscriptionConsumer() {
+	<-bc.wait // wait for our first piece of work
+
+	// the subscriptionConsumer ensures we will get nil right away if no new subscriptions is available
+	for newSubscriptions := range bc.newSubscriptions {
+		bc.updateSubscriptions(newSubscriptions)
+
+		if len(bc.subscriptions) == 0 {
+			// We're about to be shut down or we're about to receive more subscriptions.
+			// Either way, the signal just hasn't propagated to our goroutine yet.
+			<-bc.wait
+			continue
+		}
+
+		response, err := bc.fetchNewMessages()
+
+		if err != nil {
+			Logger.Printf("consumer/broker/%d disconnecting due to error processing FetchRequest: %s\n", bc.broker.ID(), err)
+			bc.abort(err)
+			return
+		}
+
+		bc.acks.Add(len(bc.subscriptions))
+		for child := range bc.subscriptions {
+			child.feeder <- response
+		}
+		bc.acks.Wait()
+		bc.handleResponses()
+	}
+}
+
+func (bc *brokerConsumer) updateSubscriptions(newSubscriptions []*partitionConsumer) {
+	for _, child := range newSubscriptions {
+		bc.subscriptions[child] = none{}
+		Logger.Printf("consumer/broker/%d added subscription to %s/%d\n", bc.broker.ID(), child.topic, child.partition)
+	}
+
+	for child := range bc.subscriptions {
+		select {
+		case <-child.dying:
+			Logger.Printf("consumer/broker/%d closed dead subscription to %s/%d\n", bc.broker.ID(), child.topic, child.partition)
+			close(child.trigger)
+			delete(bc.subscriptions, child)
+		default:
+			break
+		}
+	}
+}
+
+func (bc *brokerConsumer) handleResponses() {
+	// handles the response codes left for us by our subscriptions, and abandons ones that have been closed
+	for child := range bc.subscriptions {
+		result := child.responseResult
+		child.responseResult = nil
+
+		switch result {
+		case nil:
+			break
+		case errTimedOut:
+			Logger.Printf("consumer/broker/%d abandoned subscription to %s/%d because consuming was taking too long\n",
+				bc.broker.ID(), child.topic, child.partition)
+			delete(bc.subscriptions, child)
+		case ErrOffsetOutOfRange:
+			// there's no point in retrying this it will just fail the same way again
+			// shut it down and force the user to choose what to do
+			child.sendError(result)
+			Logger.Printf("consumer/%s/%d shutting down because %s\n", child.topic, child.partition, result)
+			close(child.trigger)
+			delete(bc.subscriptions, child)
+		case ErrUnknownTopicOrPartition, ErrNotLeaderForPartition, ErrLeaderNotAvailable, ErrReplicaNotAvailable:
+			// not an error, but does need redispatching
+			Logger.Printf("consumer/broker/%d abandoned subscription to %s/%d because %s\n",
+				bc.broker.ID(), child.topic, child.partition, result)
+			child.trigger <- none{}
+			delete(bc.subscriptions, child)
+		default:
+			// dunno, tell the user and try redispatching
+			child.sendError(result)
+			Logger.Printf("consumer/broker/%d abandoned subscription to %s/%d because %s\n",
+				bc.broker.ID(), child.topic, child.partition, result)
+			child.trigger <- none{}
+			delete(bc.subscriptions, child)
+		}
+	}
+}
+
+func (bc *brokerConsumer) abort(err error) {
+	bc.consumer.abandonBrokerConsumer(bc)
+	_ = bc.broker.Close() // we don't care about the error this might return, we already have one
+
+	for child := range bc.subscriptions {
+		child.sendError(err)
+		child.trigger <- none{}
+	}
+
+	for newSubscriptions := range bc.newSubscriptions {
+		if len(newSubscriptions) == 0 {
+			<-bc.wait
+			continue
+		}
+		for _, child := range newSubscriptions {
+			child.sendError(err)
+			child.trigger <- none{}
+		}
+	}
+}
+
+func (bc *brokerConsumer) fetchNewMessages() (*FetchResponse, error) {
+	request := &FetchRequest{
+		MinBytes:    bc.consumer.conf.Consumer.Fetch.Min,
+		MaxWaitTime: int32(bc.consumer.conf.Consumer.MaxWaitTime / time.Millisecond),
+	}
+	if bc.consumer.conf.Version.IsAtLeast(V0_10_0_0) {
+		request.Version = 2
+	}
+
+	for child := range bc.subscriptions {
+		request.AddBlock(child.topic, child.partition, child.offset, child.fetchSize)
+	}
+
+	return bc.broker.Fetch(request)
+}
@@ -0,0 +1,94 @@
+package sarama
+
+type ConsumerGroupMemberMetadata struct {
+	Version  int16
+	Topics   []string
+	UserData []byte
+}
+
+func (m *ConsumerGroupMemberMetadata) encode(pe packetEncoder) error {
+	pe.putInt16(m.Version)
+
+	if err := pe.putStringArray(m.Topics); err != nil {
+		return err
+	}
+
+	if err := pe.putBytes(m.UserData); err != nil {
+		return err
+	}
+
+	return nil
+}
+
+func (m *ConsumerGroupMemberMetadata) decode(pd packetDecoder) (err error) {
+	if m.Version, err = pd.getInt16(); err != nil {
+		return
+	}
+
+	if m.Topics, err = pd.getStringArray(); err != nil {
+		return
+	}
+
+	if m.UserData, err = pd.getBytes(); err != nil {
+		return
+	}
+
+	return nil
+}
+
+type ConsumerGroupMemberAssignment struct {
+	Version  int16
+	Topics   map[string][]int32
+	UserData []byte
+}
+
+func (m *ConsumerGroupMemberAssignment) encode(pe packetEncoder) error {
+	pe.putInt16(m.Version)
+
+	if err := pe.putArrayLength(len(m.Topics)); err != nil {
+		return err
+	}
+
+	for topic, partitions := range m.Topics {
+		if err := pe.putString(topic); err != nil {
+			return err
+		}
+		if err := pe.putInt32Array(partitions); err != nil {
+			return err
+		}
+	}
+
+	if err := pe.putBytes(m.UserData); err != nil {
+		return err
+	}
+
+	return nil
+}
+
+func (m *ConsumerGroupMemberAssignment) decode(pd packetDecoder) (err error) {
+	if m.Version, err = pd.getInt16(); err != nil {
+		return
+	}
+
+	var topicLen int
+	if topicLen, err = pd.getArrayLength(); err != nil {
+		return
+	}
+
+	m.Topics = make(map[string][]int32, topicLen)
+	for i := 0; i < topicLen; i++ {
+		var topic string
+		if topic, err = pd.getString(); err != nil {
+			return
+		}
+		if m.Topics[topic], err = pd.getInt32Array(); err != nil {
+			return
+		}
+	}
+
+	if m.UserData, err = pd.getBytes(); err != nil {
+		return
+	}
+
+	return nil
+}
@@ -0,0 +1,26 @@
+package sarama
+
+type ConsumerMetadataRequest struct {
+	ConsumerGroup string
+}
+
+func (r *ConsumerMetadataRequest) encode(pe packetEncoder) error {
+	return pe.putString(r.ConsumerGroup)
+}
+
+func (r *ConsumerMetadataRequest) decode(pd packetDecoder, version int16) (err error) {
+	r.ConsumerGroup, err = pd.getString()
+	return err
+}
+
+func (r *ConsumerMetadataRequest) key() int16 {
+	return 10
+}
+
+func (r *ConsumerMetadataRequest) version() int16 {
+	return 0
+}
+
+func (r *ConsumerMetadataRequest) requiredVersion() KafkaVersion {
+	return V0_8_2_0
+}
@@ -0,0 +1,85 @@
+package sarama
+
+import (
+	"net"
+	"strconv"
+)
+
+type ConsumerMetadataResponse struct {
+	Err             KError
+	Coordinator     *Broker
+	CoordinatorID   int32  // deprecated: use Coordinator.ID()
+	CoordinatorHost string // deprecated: use Coordinator.Addr()
+	CoordinatorPort int32  // deprecated: use Coordinator.Addr()
+}
+
+func (r *ConsumerMetadataResponse) decode(pd packetDecoder, version int16) (err error) {
+	tmp, err := pd.getInt16()
+	if err != nil {
+		return err
+	}
+	r.Err = KError(tmp)
+
+	coordinator := new(Broker)
+	if err := coordinator.decode(pd); err != nil {
+		return err
+	}
+	if coordinator.addr == ":0" {
+		return nil
+	}
+	r.Coordinator = coordinator
+
+	// this can all go away in 2.0, but we have to fill in deprecated fields to maintain
+	// backwards compatibility
+	host, portstr, err := net.SplitHostPort(r.Coordinator.Addr())
+	if err != nil {
+		return err
+	}
+	port, err := strconv.ParseInt(portstr, 10, 32)
+	if err != nil {
+		return err
+	}
+	r.CoordinatorID = r.Coordinator.ID()
+	r.CoordinatorHost = host
+	r.CoordinatorPort = int32(port)
+
+	return nil
+}
+
+func (r *ConsumerMetadataResponse) encode(pe packetEncoder) error {
+	pe.putInt16(int16(r.Err))
+	if r.Coordinator != nil {
+		host, portstr, err := net.SplitHostPort(r.Coordinator.Addr())
+		if err != nil {
+			return err
+		}
+		port, err := strconv.ParseInt(portstr, 10, 32)
+		if err != nil {
+			return err
+		}
+		pe.putInt32(r.Coordinator.ID())
+		if err := pe.putString(host); err != nil {
+			return err
+		}
+		pe.putInt32(int32(port))
+		return nil
+	}
+	pe.putInt32(r.CoordinatorID)
+	if err := pe.putString(r.CoordinatorHost); err != nil {
+		return err
+	}
+	pe.putInt32(r.CoordinatorPort)
+	return nil
+}
+
+func (r *ConsumerMetadataResponse) key() int16 {
+	return 10
+}
+
+func (r *ConsumerMetadataResponse) version() int16 {
+	return 0
+}
+
+func (r *ConsumerMetadataResponse) requiredVersion() KafkaVersion {
+	return V0_8_2_0
+}
@@ -0,0 +1,36 @@
+package sarama
+
+import (
+	"encoding/binary"
+
+	"github.com/klauspost/crc32"
+)
+
+// crc32Field implements the pushEncoder and pushDecoder interfaces for calculating CRC32s.
+type crc32Field struct {
+	startOffset int
+}
+
+func (c *crc32Field) saveOffset(in int) {
+	c.startOffset = in
+}
+
+func (c *crc32Field) reserveLength() int {
+	return 4
+}
+
+func (c *crc32Field) run(curOffset int, buf []byte) error {
+	crc := crc32.ChecksumIEEE(buf[c.startOffset+4 : curOffset])
+	binary.BigEndian.PutUint32(buf[c.startOffset:], crc)
+	return nil
+}
+
+func (c *crc32Field) check(curOffset int, buf []byte) error {
+	crc := crc32.ChecksumIEEE(buf[c.startOffset+4 : curOffset])
+
+	if crc != binary.BigEndian.Uint32(buf[c.startOffset:]) {
+		return PacketDecodingError{"CRC didn't match"}
+	}
+
+	return nil
+}
@@ -0,0 +1,30 @@
+package sarama
+
+type DescribeGroupsRequest struct {
+	Groups []string
+}
+
+func (r *DescribeGroupsRequest) encode(pe packetEncoder) error {
+	return pe.putStringArray(r.Groups)
+}
+
+func (r *DescribeGroupsRequest) decode(pd packetDecoder, version int16) (err error) {
+	r.Groups, err = pd.getStringArray()
+	return
+}
+
+func (r *DescribeGroupsRequest) key() int16 {
+	return 15
+}
+
+func (r *DescribeGroupsRequest) version() int16 {
+	return 0
+}
+
+func (r *DescribeGroupsRequest) requiredVersion() KafkaVersion {
+	return V0_9_0_0
+}
+
+func (r *DescribeGroupsRequest) AddGroup(group string) {
+	r.Groups = append(r.Groups, group)
+}
@@ -0,0 +1,186 @@
+package sarama
+
+type DescribeGroupsResponse struct {
+	Groups []*GroupDescription
+}
+
+func (r *DescribeGroupsResponse) encode(pe packetEncoder) error {
+	if err := pe.putArrayLength(len(r.Groups)); err != nil {
+		return err
+	}
+
+	for _, groupDescription := range r.Groups {
+		if err := groupDescription.encode(pe); err != nil {
+			return err
+		}
+	}
+
+	return nil
+}
+
+func (r *DescribeGroupsResponse) decode(pd packetDecoder, version int16) (err error) {
+	n, err := pd.getArrayLength()
+	if err != nil {
+		return err
+	}
+
+	r.Groups = make([]*GroupDescription, n)
+	for i := 0; i < n; i++ {
+		r.Groups[i] = new(GroupDescription)
+		if err := r.Groups[i].decode(pd); err != nil {
+			return err
+		}
+	}
+
+	return nil
+}
+
+func (r *DescribeGroupsResponse) key() int16 {
+	return 15
+}
+
+func (r *DescribeGroupsResponse) version() int16 {
+	return 0
+}
+
+func (r *DescribeGroupsResponse) requiredVersion() KafkaVersion {
+	return V0_9_0_0
+}
+
+type GroupDescription struct {
+	Err          KError
+	GroupId      string
+	State        string
+	ProtocolType string
+	Protocol     string
+	Members      map[string]*GroupMemberDescription
+}
+
+func (gd *GroupDescription) encode(pe packetEncoder) error {
+	pe.putInt16(int16(gd.Err))
+
+	if err := pe.putString(gd.GroupId); err != nil {
+		return err
+	}
+	if err := pe.putString(gd.State); err != nil {
+		return err
+	}
+	if err := pe.putString(gd.ProtocolType); err != nil {
+		return err
+	}
+	if err := pe.putString(gd.Protocol); err != nil {
+		return err
+	}
+
+	if err := pe.putArrayLength(len(gd.Members)); err != nil {
+		return err
+	}
+
+	for memberId, groupMemberDescription := range gd.Members {
+		if err := pe.putString(memberId); err != nil {
+			return err
+		}
+		if err := groupMemberDescription.encode(pe); err != nil {
+			return err
+		}
+	}
+
+	return nil
+}
+
+func (gd *GroupDescription) decode(pd packetDecoder) (err error) {
+	if kerr, err := pd.getInt16(); err != nil {
+		return err
+	} else {
+		gd.Err = KError(kerr)
+	}
+
+	if gd.GroupId, err = pd.getString(); err != nil {
+		return
+	}
+	if gd.State, err = pd.getString(); err != nil {
+		return
+	}
+	if gd.ProtocolType, err = pd.getString(); err != nil {
+		return
+	}
+	if gd.Protocol, err = pd.getString(); err != nil {
+		return
+	}
+
+	n, err := pd.getArrayLength()
+	if err != nil {
+		return err
+	}
+	if n == 0 {
+		return nil
+	}
+
+	gd.Members = make(map[string]*GroupMemberDescription)
+	for i := 0; i < n; i++ {
+		memberId, err := pd.getString()
+		if err != nil {
+			return err
+		}
+
+		gd.Members[memberId] = new(GroupMemberDescription)
+		if err := gd.Members[memberId].decode(pd); err != nil {
+			return err
+		}
+	}
+
+	return nil
+}
+
+type GroupMemberDescription struct {
+	ClientId         string
+	ClientHost       string
+	MemberMetadata   []byte
+	MemberAssignment []byte
+}
+
+func (gmd *GroupMemberDescription) encode(pe packetEncoder) error {
+	if err := pe.putString(gmd.ClientId); err != nil {
+		return err
+	}
+	if err := pe.putString(gmd.ClientHost); err != nil {
+		return err
+	}
+	if err := pe.putBytes(gmd.MemberMetadata); err != nil {
+		return err
+	}
+	if err := pe.putBytes(gmd.MemberAssignment); err != nil {
+		return err
+	}
+
+	return nil
+}
+
+func (gmd *GroupMemberDescription) decode(pd packetDecoder) (err error) {
+	if gmd.ClientId, err = pd.getString(); err != nil {
+		return
+	}
+	if gmd.ClientHost, err = pd.getString(); err != nil {
+		return
+	}
+	if gmd.MemberMetadata, err = pd.getBytes(); err != nil {
+		return
+	}
+	if gmd.MemberAssignment, err = pd.getBytes(); err != nil {
+		return
+	}
+
+	return nil
+}
+
+func (gmd *GroupMemberDescription) GetMemberAssignment() (*ConsumerGroupMemberAssignment, error) {
+	assignment := new(ConsumerGroupMemberAssignment)
+	err := decode(gmd.MemberAssignment, assignment)
+	return assignment, err
+}
+
+func (gmd *GroupMemberDescription) GetMemberMetadata() (*ConsumerGroupMemberMetadata, error) {
+	metadata := new(ConsumerGroupMemberMetadata)
+	err := decode(gmd.MemberMetadata, metadata)
+	return metadata, err
+}
@@ -0,0 +1,13 @@
+name: sarama
+
+up:
+  - go: 1.7.3
+
+commands:
+  test:
+    run: make test
+    desc: 'run unit tests'
+
+packages:
+  - git@github.com:Shopify/dev-shopify.git
+
@@ -0,0 +1,89 @@
+package sarama
+
+import (
+	"fmt"
+
+	"github.com/rcrowley/go-metrics"
+)
+
+// Encoder is the interface that wraps the basic Encode method.
+// Anything implementing Encoder can be turned into bytes using Kafka's encoding rules.
+type encoder interface {
+	encode(pe packetEncoder) error
+}
+
+// Encode takes an Encoder and turns it into bytes while potentially recording metrics.
+func encode(e encoder, metricRegistry metrics.Registry) ([]byte, error) {
+	if e == nil {
+		return nil, nil
+	}
+
+	var prepEnc prepEncoder
+	var realEnc realEncoder
+
+	err := e.encode(&prepEnc)
+	if err != nil {
+		return nil, err
+	}
+
+	if prepEnc.length < 0 || prepEnc.length > int(MaxRequestSize) {
+		return nil, PacketEncodingError{fmt.Sprintf("invalid request size (%d)", prepEnc.length)}
+	}
+
+	realEnc.raw = make([]byte, prepEnc.length)
+	realEnc.registry = metricRegistry
+	err = e.encode(&realEnc)
+	if err != nil {
+		return nil, err
+	}
+
+	return realEnc.raw, nil
+}
+
+// Decoder is the interface that wraps the basic Decode method.
+// Anything implementing Decoder can be extracted from bytes using Kafka's encoding rules.
+type decoder interface {
+	decode(pd packetDecoder) error
+}
+
+type versionedDecoder interface {
+	decode(pd packetDecoder, version int16) error
+}
+
+// Decode takes bytes and a Decoder and fills the fields of the decoder from the bytes,
+// interpreted using Kafka's encoding rules.
+func decode(buf []byte, in decoder) error {
+	if buf == nil {
+		return nil
+	}
+
+	helper := realDecoder{raw: buf}
+	err := in.decode(&helper)
+	if err != nil {
+		return err
+	}
+
+	if helper.off != len(buf) {
+		return PacketDecodingError{"invalid length"}
+	}
+
+	return nil
+}
+
+func versionedDecode(buf []byte, in versionedDecoder, version int16) error {
+	if buf == nil {
+		return nil
+	}
+
+	helper := realDecoder{raw: buf}
+	err := in.decode(&helper, version)
+	if err != nil {
+		return err
+	}
+
+	if helper.off != len(buf) {
+		return PacketDecodingError{"invalid length"}
+	}
+
+	return nil
+}
@@ -0,0 +1,197 @@
+package sarama
+
+import (
+	"errors"
+	"fmt"
+)
+
+// ErrOutOfBrokers is the error returned when the client has run out of brokers to talk to because all of them errored
+// or otherwise failed to respond.
+var ErrOutOfBrokers = errors.New("kafka: client has run out of available brokers to talk to (Is your cluster reachable?)")
+
+// ErrClosedClient is the error returned when a method is called on a client that has been closed.
+var ErrClosedClient = errors.New("kafka: tried to use a client that was closed")
+
+// ErrIncompleteResponse is the error returned when the server returns a syntactically valid response, but it does
+// not contain the expected information.
+var ErrIncompleteResponse = errors.New("kafka: response did not contain all the expected topic/partition blocks")
+
+// ErrInvalidPartition is the error returned when a partitioner returns an invalid partition index
+// (meaning one outside of the range [0...numPartitions-1]).
+var ErrInvalidPartition = errors.New("kafka: partitioner returned an invalid partition index")
+
+// ErrAlreadyConnected is the error returned when calling Open() on a Broker that is already connected or connecting.
+var ErrAlreadyConnected = errors.New("kafka: broker connection already initiated")
+
+// ErrNotConnected is the error returned when trying to send or call Close() on a Broker that is not connected.
+var ErrNotConnected = errors.New("kafka: broker not connected")
+
+// ErrInsufficientData is returned when decoding and the packet is truncated. This can be expected
+// when requesting messages, since as an optimization the server is allowed to return a partial message at the end
+// of the message set.
+var ErrInsufficientData = errors.New("kafka: insufficient data to decode packet, more bytes expected")
+
+// ErrShuttingDown is returned when a producer receives a message during shutdown.
+var ErrShuttingDown = errors.New("kafka: message received by producer in process of shutting down")
+
+// ErrMessageTooLarge is returned when the next message to consume is larger than the configured Consumer.Fetch.Max
+var ErrMessageTooLarge = errors.New("kafka: message is larger than Consumer.Fetch.Max")
+
+// PacketEncodingError is returned from a failure while encoding a Kafka packet. This can happen, for example,
+// if you try to encode a string over 2^15 characters in length, since Kafka's encoding rules do not permit that.
+type PacketEncodingError struct {
+	Info string
+}
+
+func (err PacketEncodingError) Error() string {
+	return fmt.Sprintf("kafka: error encoding packet: %s", err.Info)
+}
+
+// PacketDecodingError is returned when there was an error (other than truncated data) decoding the Kafka broker's response.
+// This can be a bad CRC or length field, or any other invalid value.
+type PacketDecodingError struct {
+	Info string
+}
+
+func (err PacketDecodingError) Error() string {
+	return fmt.Sprintf("kafka: error decoding packet: %s", err.Info)
+}
+
+// ConfigurationError is the type of error returned from a constructor (e.g. NewClient, or NewConsumer)
+// when the specified configuration is invalid.
+type ConfigurationError string
+
+func (err ConfigurationError) Error() string {
+	return "kafka: invalid configuration (" + string(err) + ")"
+}
+
+// KError is the type of error that can be returned directly by the Kafka broker.
+// See https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol#AGuideToTheKafkaProtocol-ErrorCodes
+type KError int16
+
+// Numeric error codes returned by the Kafka server.
+const (
+	ErrNoError                         KError = 0
+	ErrUnknown                         KError = -1
+	ErrOffsetOutOfRange                KError = 1
+	ErrInvalidMessage                  KError = 2
+	ErrUnknownTopicOrPartition         KError = 3
+	ErrInvalidMessageSize              KError = 4
+	ErrLeaderNotAvailable              KError = 5
+	ErrNotLeaderForPartition           KError = 6
+	ErrRequestTimedOut                 KError = 7
+	ErrBrokerNotAvailable              KError = 8
+	ErrReplicaNotAvailable             KError = 9
+	ErrMessageSizeTooLarge             KError = 10
+	ErrStaleControllerEpochCode        KError = 11
+	ErrOffsetMetadataTooLarge          KError = 12
+	ErrNetworkException                KError = 13
+	ErrOffsetsLoadInProgress           KError = 14
+	ErrConsumerCoordinatorNotAvailable KError = 15
+	ErrNotCoordinatorForConsumer       KError = 16
+	ErrInvalidTopic                    KError = 17
+	ErrMessageSetSizeTooLarge          KError = 18
+	ErrNotEnoughReplicas               KError = 19
+	ErrNotEnoughReplicasAfterAppend    KError = 20
+	ErrInvalidRequiredAcks             KError = 21
+	ErrIllegalGeneration               KError = 22
+	ErrInconsistentGroupProtocol       KError = 23
+	ErrInvalidGroupId                  KError = 24
+	ErrUnknownMemberId                 KError = 25
+	ErrInvalidSessionTimeout           KError = 26
+	ErrRebalanceInProgress             KError = 27
+	ErrInvalidCommitOffsetSize         KError = 28
+	ErrTopicAuthorizationFailed        KError = 29
+	ErrGroupAuthorizationFailed        KError = 30
+	ErrClusterAuthorizationFailed      KError = 31
+	ErrInvalidTimestamp                KError = 32
+	ErrUnsupportedSASLMechanism        KError = 33
+	ErrIllegalSASLState                KError = 34
+	ErrUnsupportedVersion              KError = 35
+	ErrUnsupportedForMessageFormat     KError = 43
+)
+
+func (err KError) Error() string {
+	// Error messages stolen/adapted from
+	// https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol
+	switch err {
+	case ErrNoError:
+		return "kafka server: Not an error, why are you printing me?"
+	case ErrUnknown:
+		return "kafka server: Unexpected (unknown?) server error."
+	case ErrOffsetOutOfRange:
+		return "kafka server: The requested offset is outside the range of offsets maintained by the server for the given topic/partition."
+	case ErrInvalidMessage:
+		return "kafka server: Message contents does not match its CRC."
+	case ErrUnknownTopicOrPartition:
+		return "kafka server: Request was for a topic or partition that does not exist on this broker."
+	case ErrInvalidMessageSize:
+		return "kafka server: The message has a negative size."
+	case ErrLeaderNotAvailable:
+		return "kafka server: In the middle of a leadership election, there is currently no leader for this partition and hence it is unavailable for writes."
+	case ErrNotLeaderForPartition:
+		return "kafka server: Tried to send a message to a replica that is not the leader for some partition. Your metadata is out of date."
+	case ErrRequestTimedOut:
+		return "kafka server: Request exceeded the user-specified time limit in the request."
+	case ErrBrokerNotAvailable:
+		return "kafka server: Broker not available. Not a client facing error, we should never receive this!!!"
+	case ErrReplicaNotAvailable:
+		return "kafka server: Replica information not available, one or more brokers are down."
+	case ErrMessageSizeTooLarge:
+		return "kafka server: Message was too large, server rejected it to avoid allocation error."
+	case ErrStaleControllerEpochCode:
+		return "kafka server: StaleControllerEpochCode (internal error code for broker-to-broker communication)."
+	case ErrOffsetMetadataTooLarge:
+		return "kafka server: Specified a string larger than the configured maximum for offset metadata."
+	case ErrNetworkException:
+		return "kafka server: The server disconnected before a response was received."
+	case ErrOffsetsLoadInProgress:
+		return "kafka server: The broker is still loading offsets after a leader change for that offset's topic partition."
+	case ErrConsumerCoordinatorNotAvailable:
+		return "kafka server: Offset's topic has not yet been created."
+	case ErrNotCoordinatorForConsumer:
+		return "kafka server: Request was for a consumer group that is not coordinated by this broker."
+	case ErrInvalidTopic:
+		return "kafka server: The request attempted to perform an operation on an invalid topic."
+	case ErrMessageSetSizeTooLarge:
+		return "kafka server: The request included message batch larger than the configured segment size on the server."
+	case ErrNotEnoughReplicas:
+		return "kafka server: Messages are rejected since there are fewer in-sync replicas than required."
+	case ErrNotEnoughReplicasAfterAppend:
+		return "kafka server: Messages are written to the log, but to fewer in-sync replicas than required."
+	case ErrInvalidRequiredAcks:
+		return "kafka server: The number of required acks is invalid (should be either -1, 0, or 1)."
+	case ErrIllegalGeneration:
+		return "kafka server: The provided generation id is not the current generation."
+	case ErrInconsistentGroupProtocol:
+		return "kafka server: The provider group protocol type is incompatible with the other members."
+	case ErrInvalidGroupId:
+		return "kafka server: The provided group id was empty."
+	case ErrUnknownMemberId:
+		return "kafka server: The provided member is not known in the current generation."
+	case ErrInvalidSessionTimeout:
+		return "kafka server: The provided session timeout is outside the allowed range."
+	case ErrRebalanceInProgress:
+		return "kafka server: A rebalance for the group is in progress. Please re-join the group."
+	case ErrInvalidCommitOffsetSize:
+		return "kafka server: The provided commit metadata was too large."
+	case ErrTopicAuthorizationFailed:
+		return "kafka server: The client is not authorized to access this topic."
+	case ErrGroupAuthorizationFailed:
+		return "kafka server: The client is not authorized to access this group."
+	case ErrClusterAuthorizationFailed:
+		return "kafka server: The client is not authorized to send this request type."
+	case ErrInvalidTimestamp:
+		return "kafka server: The timestamp of the message is out of acceptable range."
+	case ErrUnsupportedSASLMechanism:
+		return "kafka server: The broker does not support the requested SASL mechanism."
+	case ErrIllegalSASLState:
+		return "kafka server: Request is not valid given the current SASL state."
+	case ErrUnsupportedVersion:
+		return "kafka server: The version of API is not supported."
+	case ErrUnsupportedForMessageFormat:
+		return "kafka server: The requested operation is not supported by the message format version."
+	}
+
+	return fmt.Sprintf("Unknown error, how did this happen? Error code = %d", err)
+}
@@ -0,0 +1,136 @@
+package sarama
+
+type fetchRequestBlock struct {
+	fetchOffset int64
+	maxBytes    int32
+}
+
+func (b *fetchRequestBlock) encode(pe packetEncoder) error {
+	pe.putInt64(b.fetchOffset)
+	pe.putInt32(b.maxBytes)
+	return nil
+}
+
+func (b *fetchRequestBlock) decode(pd packetDecoder) (err error) {
+	if b.fetchOffset, err = pd.getInt64(); err != nil {
+		return err
+	}
+	if b.maxBytes, err = pd.getInt32(); err != nil {
+		return err
+	}
+	return nil
+}
+
+type FetchRequest struct {
+	MaxWaitTime int32
+	MinBytes    int32
+	Version     int16
+	blocks      map[string]map[int32]*fetchRequestBlock
+}
+
+func (r *FetchRequest) encode(pe packetEncoder) (err error) {
+	pe.putInt32(-1) // replica ID is always -1 for clients
+	pe.putInt32(r.MaxWaitTime)
+	pe.putInt32(r.MinBytes)
+	err = pe.putArrayLength(len(r.blocks))
+	if err != nil {
+		return err
+	}
+	for topic, blocks := range r.blocks {
+		err = pe.putString(topic)
+		if err != nil {
+			return err
+		}
+		err = pe.putArrayLength(len(blocks))
+		if err != nil {
+			return err
+		}
+		for partition, block := range blocks {
+			pe.putInt32(partition)
+			err = block.encode(pe)
+			if err != nil {
+				return err
+			}
+		}
+	}
+	return nil
+}
+
+func (r *FetchRequest) decode(pd packetDecoder, version int16) (err error) {
+	r.Version = version
+	if _, err = pd.getInt32(); err != nil {
+		return err
+	}
+	if r.MaxWaitTime, err = pd.getInt32(); err != nil {
+		return err
+	}
+	if r.MinBytes, err = pd.getInt32(); err != nil {
+		return err
+	}
+	topicCount, err := pd.getArrayLength()
+	if err != nil {
+		return err
+	}
+	if topicCount == 0 {
+		return nil
+	}
+	r.blocks = make(map[string]map[int32]*fetchRequestBlock)
+	for i := 0; i < topicCount; i++ {
+		topic, err := pd.getString()
+		if err != nil {
+			return err
+		}
+		partitionCount, err := pd.getArrayLength()
+		if err != nil {
+			return err
+		}
+		r.blocks[topic] = make(map[int32]*fetchRequestBlock)
+		for j := 0; j < partitionCount; j++ {
+			partition, err := pd.getInt32()
+			if err != nil {
+				return err
+			}
+			fetchBlock := &fetchRequestBlock{}
+			if err = fetchBlock.decode(pd); err != nil {
+				return nil
+			}
+			r.blocks[topic][partition] = fetchBlock
+		}
+	}
+	return nil
+}
+
+func (r *FetchRequest) key() int16 {
+	return 1
+}
+
+func (r *FetchRequest) version() int16 {
+	return r.Version
+}
+
+func (r *FetchRequest) requiredVersion() KafkaVersion {
+	switch r.Version {
+	case 1:
+		return V0_9_0_0
+	case 2:
+		return V0_10_0_0
+	default:
+		return minVersion
+	}
+}
+
+func (r *FetchRequest) AddBlock(topic string, partitionID int32, fetchOffset int64, maxBytes int32) {
+	if r.blocks == nil {
+		r.blocks = make(map[string]map[int32]*fetchRequestBlock)
+	}
+
+	if r.blocks[topic] == nil {
+		r.blocks[topic] = make(map[int32]*fetchRequestBlock)
+	}
+
+	tmp := new(fetchRequestBlock)
+	tmp.maxBytes = maxBytes
+	tmp.fetchOffset = fetchOffset
+
+	r.blocks[topic][partitionID] = tmp
+}
@@ -0,0 +1,210 @@
+package sarama
+
+import "time"
+
+type FetchResponseBlock struct {
+	Err                 KError
+	HighWaterMarkOffset int64
+	MsgSet              MessageSet
+}
+
+func (b *FetchResponseBlock) decode(pd packetDecoder) (err error) {
+	tmp, err := pd.getInt16()
+	if err != nil {
+		return err
+	}
+	b.Err = KError(tmp)
+
+	b.HighWaterMarkOffset, err = pd.getInt64()
+	if err != nil {
+		return err
+	}
+
+	msgSetSize, err := pd.getInt32()
+	if err != nil {
+		return err
+	}
+
+	msgSetDecoder, err := pd.getSubset(int(msgSetSize))
+	if err != nil {
+		return err
+	}
+	err = (&b.MsgSet).decode(msgSetDecoder)
+
+	return err
+}
+
+func (b *FetchResponseBlock) encode(pe packetEncoder) (err error) {
+	pe.putInt16(int16(b.Err))
+
+	pe.putInt64(b.HighWaterMarkOffset)
+
+	pe.push(&lengthField{})
+	err = b.MsgSet.encode(pe)
+	if err != nil {
+		return err
+	}
+	return pe.pop()
+}
+
+type FetchResponse struct {
+	Blocks       map[string]map[int32]*FetchResponseBlock
+	ThrottleTime time.Duration
+	Version      int16 // v1 requires 0.9+, v2 requires 0.10+
+}
+
+func (r *FetchResponse) decode(pd packetDecoder, version int16) (err error) {
+	r.Version = version
+
+	if r.Version >= 1 {
+		throttle, err := pd.getInt32()
+		if err != nil {
+			return err
+		}
+		r.ThrottleTime = time.Duration(throttle) * time.Millisecond
+	}
+
+	numTopics, err := pd.getArrayLength()
+	if err != nil {
+		return err
+	}
+
+	r.Blocks = make(map[string]map[int32]*FetchResponseBlock, numTopics)
+	for i := 0; i < numTopics; i++ {
+		name, err := pd.getString()
+		if err != nil {
+			return err
+		}
+
+		numBlocks, err := pd.getArrayLength()
+		if err != nil {
+			return err
+		}
+
+		r.Blocks[name] = make(map[int32]*FetchResponseBlock, numBlocks)
+
+		for j := 0; j < numBlocks; j++ {
+			id, err := pd.getInt32()
+			if err != nil {
+				return err
+			}
+
+			block := new(FetchResponseBlock)
+			err = block.decode(pd)
+			if err != nil {
+				return err
+			}
+			r.Blocks[name][id] = block
+		}
+	}
+
+	return nil
+}
+
+func (r *FetchResponse) encode(pe packetEncoder) (err error) {
+	if r.Version >= 1 {
+		pe.putInt32(int32(r.ThrottleTime / time.Millisecond))
+	}
+
+	err = pe.putArrayLength(len(r.Blocks))
+	if err != nil {
+		return err
+	}
+
+	for topic, partitions := range r.Blocks {
+		err = pe.putString(topic)
+		if err != nil {
+			return err
+		}
+
+		err = pe.putArrayLength(len(partitions))
+		if err != nil {
+			return err
+		}
+
+		for id, block := range partitions {
+			pe.putInt32(id)
+			err = block.encode(pe)
+			if err != nil {
+				return err
+			}
+		}
+
+	}
+	return nil
+}
+
+func (r *FetchResponse) key() int16 {
+	return 1
+}
+
+func (r *FetchResponse) version() int16 {
+	return r.Version
+}
+
+func (r *FetchResponse) requiredVersion() KafkaVersion {
+	switch r.Version {
+	case 1:
+		return V0_9_0_0
+	case 2:
+		return V0_10_0_0
+	default:
+		return minVersion
+	}
+}
+
+func (r *FetchResponse) GetBlock(topic string, partition int32) *FetchResponseBlock {
+	if r.Blocks == nil {
+		return nil
+	}
+
+	if r.Blocks[topic] == nil {
+		return nil
+	}
+
+	return r.Blocks[topic][partition]
+}
+
+func (r *FetchResponse) AddError(topic string, partition int32, err KError) {
+	if r.Blocks == nil {
+		r.Blocks = make(map[string]map[int32]*FetchResponseBlock)
+	}
+	partitions, ok := r.Blocks[topic]
+	if !ok {
+		partitions = make(map[int32]*FetchResponseBlock)
+		r.Blocks[topic] = partitions
+	}
+	frb, ok := partitions[partition]
+	if !ok {
+		frb = new(FetchResponseBlock)
+		partitions[partition] = frb
+	}
+	frb.Err = err
+}
+
+func (r *FetchResponse) AddMessage(topic string, partition int32, key, value Encoder, offset int64) {
+	if r.Blocks == nil {
+		r.Blocks = make(map[string]map[int32]*FetchResponseBlock)
+	}
+	partitions, ok := r.Blocks[topic]
+	if !ok {
+		partitions = make(map[int32]*FetchResponseBlock)
+		r.Blocks[topic] = partitions
+	}
+	frb, ok := partitions[partition]
+	if !ok {
+		frb = new(FetchResponseBlock)
+		partitions[partition] = frb
+	}
+	var kb []byte
+	var vb []byte
+	if key != nil {
+		kb, _ = key.Encode()
+	}
+	if value != nil {
+		vb, _ = value.Encode()
+	}
+	msg := &Message{Key: kb, Value: vb}
+	msgBlock := &MessageBlock{Msg: msg, Offset: offset}
+	frb.MsgSet.Messages = append(frb.MsgSet.Messages, msgBlock)
+}
@@ -0,0 +1,47 @@
+package sarama
+
+type HeartbeatRequest struct {
+	GroupId      string
+	GenerationId int32
+	MemberId     string
+}
+
+func (r *HeartbeatRequest) encode(pe packetEncoder) error {
+	if err := pe.putString(r.GroupId); err != nil {
+		return err
+	}
+
+	pe.putInt32(r.GenerationId)
+
+	if err := pe.putString(r.MemberId); err != nil {
+		return err
+	}
+
+	return nil
+}
+
+func (r *HeartbeatRequest) decode(pd packetDecoder, version int16) (err error) {
+	if r.GroupId, err = pd.getString(); err != nil {
+		return
+	}
+	if r.GenerationId, err = pd.getInt32(); err != nil {
+		return
+	}
+	if r.MemberId, err = pd.getString(); err != nil {
+		return
+	}
+
+	return nil
+}
+
+func (r *HeartbeatRequest) key() int16 {
+	return 12
+}
+
+func (r *HeartbeatRequest) version() int16 {
+	return 0
+}
+
+func (r *HeartbeatRequest) requiredVersion() KafkaVersion {
+	return V0_9_0_0
+}
@@ -0,0 +1,32 @@
+package sarama
+
+type HeartbeatResponse struct {
+	Err KError
+}
+
+func (r *HeartbeatResponse) encode(pe packetEncoder) error {
+	pe.putInt16(int16(r.Err))
+	return nil
+}
+
+func (r *HeartbeatResponse) decode(pd packetDecoder, version int16) error {
+	if kerr, err := pd.getInt16(); err != nil {
+		return err
+	} else {
+		r.Err = KError(kerr)
+	}
+
+	return nil
+}
+
+func (r *HeartbeatResponse) key() int16 {
+	return 12
+}
+
+func (r *HeartbeatResponse) version() int16 {
+	return 0
+}
+
+func (r *HeartbeatResponse) requiredVersion() KafkaVersion {
+	return V0_9_0_0
+}
@@ -0,0 +1,108 @@
+package sarama
+
+type JoinGroupRequest struct {
+	GroupId        string
+	SessionTimeout int32
+	MemberId       string
+	ProtocolType   string
+	GroupProtocols map[string][]byte
+}
+
+func (r *JoinGroupRequest) encode(pe packetEncoder) error {
+	if err := pe.putString(r.GroupId); err != nil {
+		return err
+	}
+	pe.putInt32(r.SessionTimeout)
+	if err := pe.putString(r.MemberId); err != nil {
+		return err
+	}
+	if err := pe.putString(r.ProtocolType); err != nil {
+		return err
+	}
+
+	if err := pe.putArrayLength(len(r.GroupProtocols)); err != nil {
+		return err
+	}
+	for name, metadata := range r.GroupProtocols {
+		if err := pe.putString(name); err != nil {
+			return err
+		}
+		if err := pe.putBytes(metadata); err != nil {
+			return err
+		}
+	}
+
+	return nil
+}
+
+func (r *JoinGroupRequest) decode(pd packetDecoder, version int16) (err error) {
+	if r.GroupId, err = pd.getString(); err != nil {
+		return
+	}
+
+	if r.SessionTimeout, err = pd.getInt32(); err != nil {
+		return
+	}
+
+	if r.MemberId, err = pd.getString(); err != nil {
+		return
+	}
+
+	if r.ProtocolType, err = pd.getString(); err != nil {
+		return
+	}
+
+	n, err := pd.getArrayLength()
+	if err != nil {
+		return err
+	}
+	if n == 0 {
+		return nil
+	}
+
+	r.GroupProtocols = make(map[string][]byte)
+	for i := 0; i < n; i++ {
+		name, err := pd.getString()
+		if err != nil {
+			return err
+		}
+		metadata, err := pd.getBytes()
+		if err != nil {
+			return err
+		}
+
+		r.GroupProtocols[name] = metadata
+	}
+
+	return nil
+}
+
+func (r *JoinGroupRequest) key() int16 {
+	return 11
+}
+
+func (r *JoinGroupRequest) version() int16 {
+	return 0
+}
+
+func (r *JoinGroupRequest) requiredVersion() KafkaVersion {
+	return V0_9_0_0
+}
+
+func (r *JoinGroupRequest) AddGroupProtocol(name string, metadata []byte) {
+	if r.GroupProtocols == nil {
+		r.GroupProtocols = make(map[string][]byte)
+	}
+
+	r.GroupProtocols[name] = metadata
+}
+
+func (r *JoinGroupRequest) AddGroupProtocolMetadata(name string, metadata *ConsumerGroupMemberMetadata) error {
+	bin, err := encode(metadata, nil)
+	if err != nil {
+		return err
+	}
+
+	r.AddGroupProtocol(name, bin)
+	return nil
+}
@@ -0,0 +1,114 @@
+package sarama
+
+type JoinGroupResponse struct {
+	Err           KError
+	GenerationId  int32
+	GroupProtocol string
+	LeaderId      string
+	MemberId      string
+	Members       map[string][]byte
+}
+
+func (r *JoinGroupResponse) GetMembers() (map[string]ConsumerGroupMemberMetadata, error) {
+	members := make(map[string]ConsumerGroupMemberMetadata, len(r.Members))
+	for id, bin := range r.Members {
+		meta := new(ConsumerGroupMemberMetadata)
+		if err := decode(bin, meta); err != nil {
+			return nil, err
+		}
+		members[id] = *meta
+	}
+	return members, nil
+}
+
+func (r *JoinGroupResponse) encode(pe packetEncoder) error {
+	pe.putInt16(int16(r.Err))
+	pe.putInt32(r.GenerationId)
+
+	if err := pe.putString(r.GroupProtocol); err != nil {
+		return err
+	}
+	if err := pe.putString(r.LeaderId); err != nil {
+		return err
+	}
+	if err := pe.putString(r.MemberId); err != nil {
+		return err
+	}
+
+	if err := pe.putArrayLength(len(r.Members)); err != nil {
+		return err
+	}
+
+	for memberId, memberMetadata := range r.Members {
+		if err := pe.putString(memberId); err != nil {
+			return err
+		}
+
+		if err := pe.putBytes(memberMetadata); err != nil {
+			return err
+		}
+	}
+
+	return nil
+}
+
+func (r *JoinGroupResponse) decode(pd packetDecoder, version int16) (err error) {
+	if kerr, err := pd.getInt16(); err != nil {
+		return err
+	} else {
+		r.Err = KError(kerr)
+	}
+
+	if r.GenerationId, err = pd.getInt32(); err != nil {
+		return
+	}
+
+	if r.GroupProtocol, err = pd.getString(); err != nil {
+		return
+	}
+
+	if r.LeaderId, err = pd.getString(); err != nil {
+		return
+	}
+
+	if r.MemberId, err = pd.getString(); err != nil {
+		return
+	}
+
+	n, err := pd.getArrayLength()
+	if err != nil {
+		return err
+	}
+	if n == 0 {
+		return nil
+	}
+
+	r.Members = make(map[string][]byte)
+	for i := 0; i < n; i++ {
+		memberId, err := pd.getString()
+		if err != nil {
+			return err
+		}
+
+		memberMetadata, err := pd.getBytes()
+		if err != nil {
+			return err
+		}
+
+		r.Members[memberId] = memberMetadata
+	}
+
+	return nil
+}
+
+func (r *JoinGroupResponse) key() int16 {
+	return 11
+}
+
+func (r *JoinGroupResponse) version() int16 {
+	return 0
+}
+
+func (r *JoinGroupResponse) requiredVersion() KafkaVersion {
+	return V0_9_0_0
+}
@@ -0,0 +1,40 @@
+package sarama
+
+type LeaveGroupRequest struct {
+	GroupId  string
+	MemberId string
+}
+
+func (r *LeaveGroupRequest) encode(pe packetEncoder) error {
+	if err := pe.putString(r.GroupId); err != nil {
+		return err
+	}
+	if err := pe.putString(r.MemberId); err != nil {
+		return err
+	}
+
+	return nil
+}
+
+func (r *LeaveGroupRequest) decode(pd packetDecoder, version int16) (err error) {
+	if r.GroupId, err = pd.getString(); err != nil {
+		return
+	}
+	if r.MemberId, err = pd.getString(); err != nil {
+		return
+	}
+
+	return nil
+}
+
+func (r *LeaveGroupRequest) key() int16 {
+	return 13
+}
+
+func (r *LeaveGroupRequest) version() int16 {
+	return 0
+}
+
+func (r *LeaveGroupRequest) requiredVersion() KafkaVersion {
+	return V0_9_0_0
+}
@@ -0,0 +1,32 @@
+package sarama
+
+type LeaveGroupResponse struct {
+	Err KError
+}
+
+func (r *LeaveGroupResponse) encode(pe packetEncoder) error {
+	pe.putInt16(int16(r.Err))
+	return nil
+}
+
+func (r *LeaveGroupResponse) decode(pd packetDecoder, version int16) (err error) {
+	if kerr, err := pd.getInt16(); err != nil {
+		return err
+	} else {
+		r.Err = KError(kerr)
+	}
+
+	return nil
+}
+
+func (r *LeaveGroupResponse) key() int16 {
+	return 13
+}
+
+func (r *LeaveGroupResponse) version() int16 {
+	return 0
+}
+
+func (r *LeaveGroupResponse) requiredVersion() KafkaVersion {
+	return V0_9_0_0
+}
@@ -0,0 +1,29 @@
+package sarama
+
+import "encoding/binary"
+
+// LengthField implements the PushEncoder and PushDecoder interfaces for calculating 4-byte lengths.
+type lengthField struct {
+	startOffset int
+}
+
+func (l *lengthField) saveOffset(in int) {
+	l.startOffset = in
+}
+
+func (l *lengthField) reserveLength() int {
+	return 4
+}
+
+func (l *lengthField) run(curOffset int, buf []byte) error {
+	binary.BigEndian.PutUint32(buf[l.startOffset:], uint32(curOffset-l.startOffset-4))
+	return nil
+}
+
+func (l *lengthField) check(curOffset int, buf []byte) error {
+	if uint32(curOffset-l.startOffset-4) != binary.BigEndian.Uint32(buf[l.startOffset:]) {
+		return PacketDecodingError{"length field invalid"}
+	}
+
+	return nil
+}
@@ -0,0 +1,24 @@
+package sarama
+
+type ListGroupsRequest struct {
+}
+
+func (r *ListGroupsRequest) encode(pe packetEncoder) error {
+	return nil
+}
+
+func (r *ListGroupsRequest) decode(pd packetDecoder, version int16) (err error) {
+	return nil
+}
+
+func (r *ListGroupsRequest) key() int16 {
+	return 16
+}
+
+func (r *ListGroupsRequest) version() int16 {
+	return 0
+}
+
+func (r *ListGroupsRequest) requiredVersion() KafkaVersion {
+	return V0_9_0_0
+}
@@ -0,0 +1,68 @@
+package sarama
+
+type ListGroupsResponse struct {
+	Err    KError
+	Groups map[string]string
+}
+
+func (r *ListGroupsResponse) encode(pe packetEncoder) error {
+	pe.putInt16(int16(r.Err))
+
+	if err := pe.putArrayLength(len(r.Groups)); err != nil {
+		return err
+	}
+	for groupId, protocolType := range r.Groups {
+		if err := pe.putString(groupId); err != nil {
+			return err
+		}
+		if err := pe.putString(protocolType); err != nil {
+			return err
+		}
+	}
+
+	return nil
+}
+
+func (r *ListGroupsResponse) decode(pd packetDecoder, version int16) error {
+	if kerr, err := pd.getInt16(); err != nil {
+		return err
+	} else {
+		r.Err = KError(kerr)
+	}
+
+	n, err := pd.getArrayLength()
+	if err != nil {
+		return err
+	}
+	if n == 0 {
+		return nil
+	}
+
+	r.Groups = make(map[string]string)
+	for i := 0; i < n; i++ {
+		groupId, err := pd.getString()
+		if err != nil {
+			return err
+		}
+		protocolType, err := pd.getString()
+		if err != nil {
+			return err
+		}
+
+		r.Groups[groupId] = protocolType
+	}
+
+	return nil
+}
+
+func (r *ListGroupsResponse) key() int16 {
+	return 16
+}
+
+func (r *ListGroupsResponse) version() int16 {
+	return 0
+}
+
+func (r *ListGroupsResponse) requiredVersion() KafkaVersion {
+	return V0_9_0_0
+}
@@ -0,0 +1,196 @@
+package sarama
+
+import (
+	"bytes"
+	"compress/gzip"
+	"fmt"
+	"io/ioutil"
+	"time"
+
+	"github.com/eapache/go-xerial-snappy"
+	"github.com/pierrec/lz4"
+)
+
+// CompressionCodec represents the various compression codecs recognized by Kafka in messages.
+type CompressionCodec int8
+
+// only the last two bits are really used
+const compressionCodecMask int8 = 0x03
+
+const (
+	CompressionNone   CompressionCodec = 0
+	CompressionGZIP   CompressionCodec = 1
+	CompressionSnappy CompressionCodec = 2
+	CompressionLZ4    CompressionCodec = 3
+)
+
+type Message struct {
+	Codec     CompressionCodec // codec used to compress the message contents
+	Key       []byte           // the message key, may be nil
+	Value     []byte           // the message contents
+	Set       *MessageSet      // the message set a message might wrap
+	Version   int8             // v1 requires Kafka 0.10
+	Timestamp time.Time        // the timestamp of the message (version 1+ only)
+
+	compressedCache []byte
+	compressedSize  int // used for computing the compression ratio metrics
+}
+
+func (m *Message) encode(pe packetEncoder) error {
+	pe.push(&crc32Field{})
+
+	pe.putInt8(m.Version)
+
+	attributes := int8(m.Codec) & compressionCodecMask
+	pe.putInt8(attributes)
+
+	if m.Version >= 1 {
+		pe.putInt64(m.Timestamp.UnixNano() / int64(time.Millisecond))
+	}
+
+	err := pe.putBytes(m.Key)
+	if err != nil {
+		return err
+	}
+
+	var payload []byte
+
+	if m.compressedCache != nil {
+		payload = m.compressedCache
+		m.compressedCache = nil
+	} else if m.Value != nil {
+		switch m.Codec {
+		case CompressionNone:
+			payload = m.Value
+		case CompressionGZIP:
+			var buf bytes.Buffer
+			writer := gzip.NewWriter(&buf)
+			if _, err = writer.Write(m.Value); err != nil {
+				return err
+			}
+			if err = writer.Close(); err != nil {
+				return err
+			}
+			m.compressedCache = buf.Bytes()
+			payload = m.compressedCache
+		case CompressionSnappy:
+			tmp := snappy.Encode(m.Value)
+			m.compressedCache = tmp
+			payload = m.compressedCache
+		case CompressionLZ4:
+			var buf bytes.Buffer
+			writer := lz4.NewWriter(&buf)
+			if _, err = writer.Write(m.Value); err != nil {
+				return err
+			}
+			if err = writer.Close(); err != nil {
+				return err
+			}
+			m.compressedCache = buf.Bytes()
+			payload = m.compressedCache
+
+		default:
+			return PacketEncodingError{fmt.Sprintf("unsupported compression codec (%d)", m.Codec)}
+		}
+		// Keep in mind the compressed payload size for metric gathering
+		m.compressedSize = len(payload)
+	}
+
+	if err = pe.putBytes(payload); err != nil {
+		return err
+	}
+
+	return pe.pop()
+}
+
+func (m *Message) decode(pd packetDecoder) (err error) {
+	err = pd.push(&crc32Field{})
+	if err != nil {
+		return err
+	}
+
+	m.Version, err = pd.getInt8()
+	if err != nil {
+		return err
+	}
+
+	attribute, err := pd.getInt8()
+	if err != nil {
+		return err
+	}
+	m.Codec = CompressionCodec(attribute & compressionCodecMask)
+
+	if m.Version >= 1 {
+		millis, err := pd.getInt64()
+		if err != nil {
+			return err
+		}
+		m.Timestamp = time.Unix(millis/1000, (millis%1000)*int64(time.Millisecond))
+	}
+
+	m.Key, err = pd.getBytes()
+	if err != nil {
+		return err
+	}
+
+	m.Value, err = pd.getBytes()
+	if err != nil {
+		return err
+	}
+
+	// Required for deep equal assertion during tests but might be useful
+	// for future metrics about the compression ratio in fetch requests
+	m.compressedSize = len(m.Value)
+
+	switch m.Codec {
+	case CompressionNone:
+		// nothing to do
+	case CompressionGZIP:
+		if m.Value == nil {
+			break
+		}
+		reader, err := gzip.NewReader(bytes.NewReader(m.Value))
+		if err != nil {
+			return err
+		}
+		if m.Value, err = ioutil.ReadAll(reader); err != nil {
+			return err
+		}
+		if err := m.decodeSet(); err != nil {
+			return err
+		}
+	case CompressionSnappy:
+		if m.Value == nil {
+			break
+		}
+		if m.Value, err = snappy.Decode(m.Value); err != nil {
+			return err
+		}
+		if err := m.decodeSet(); err != nil {
+			return err
+		}
+	case CompressionLZ4:
+		if m.Value == nil {
+			break
+		}
+		reader := lz4.NewReader(bytes.NewReader(m.Value))
+		if m.Value, err = ioutil.ReadAll(reader); err != nil {
+			return err
+		}
+		if err := m.decodeSet(); err != nil {
+			return err
+		}
+
+	default:
+		return PacketDecodingError{fmt.Sprintf("invalid compression specified (%d)", m.Codec)}
+	}
+
+	return pd.pop()
+}
+
+// decodes a message set from a previousy encoded bulk-message
+func (m *Message) decodeSet() (err error) {
+	pd := realDecoder{raw: m.Value}
+	m.Set = &MessageSet{}
+	return m.Set.decode(&pd)
+}
@@ -0,0 +1,89 @@
+package sarama
+
+type MessageBlock struct {
+	Offset int64
+	Msg    *Message
+}
+
+// Messages convenience helper which returns either all the
+// messages that are wrapped in this block
+func (msb *MessageBlock) Messages() []*MessageBlock {
+	if msb.Msg.Set != nil {
+		return msb.Msg.Set.Messages
+	}
+	return []*MessageBlock{msb}
+}
+
+func (msb *MessageBlock) encode(pe packetEncoder) error {
+	pe.putInt64(msb.Offset)
+	pe.push(&lengthField{})
+	err := msb.Msg.encode(pe)
+	if err != nil {
+		return err
+	}
+	return pe.pop()
+}
+
+func (msb *MessageBlock) decode(pd packetDecoder) (err error) {
+	if msb.Offset, err = pd.getInt64(); err != nil {
+		return err
+	}
+
+	if err = pd.push(&lengthField{}); err != nil {
+		return err
+	}
+
+	msb.Msg = new(Message)
+	if err = msb.Msg.decode(pd); err != nil {
+		return err
+	}
+
+	if err = pd.pop(); err != nil {
+		return err
+	}
+
+	return nil
+}
+
+type MessageSet struct {
+	PartialTrailingMessage bool // whether the set on the wire contained an incomplete trailing MessageBlock
+	Messages               []*MessageBlock
+}
+
+func (ms *MessageSet) encode(pe packetEncoder) error {
+	for i := range ms.Messages {
+		err := ms.Messages[i].encode(pe)
+		if err != nil {
+			return err
+		}
+	}
+	return nil
+}
+
+func (ms *MessageSet) decode(pd packetDecoder) (err error) {
+	ms.Messages = nil
+
+	for pd.remaining() > 0 {
+		msb := new(MessageBlock)
+		err = msb.decode(pd)
+		switch err {
+		case nil:
+			ms.Messages = append(ms.Messages, msb)
+		case ErrInsufficientData:
+			// As an optimization the server is allowed to return a partial message at the
+			// end of the message set. Clients should handle this case. So we just ignore such things.
+			ms.PartialTrailingMessage = true
+			return nil
+		default:
+			return err
+		}
+	}
+
+	return nil
+}
+
+func (ms *MessageSet) addMessage(msg *Message) {
+	block := new(MessageBlock)
+	block.Msg = msg
+	ms.Messages = append(ms.Messages, block)
+}
@@ -0,0 +1,52 @@
+package sarama
+
+type MetadataRequest struct {
+	Topics []string
+}
+
+func (r *MetadataRequest) encode(pe packetEncoder) error {
+	err := pe.putArrayLength(len(r.Topics))
+	if err != nil {
+		return err
+	}
+
+	for i := range r.Topics {
+		err = pe.putString(r.Topics[i])
+		if err != nil {
+			return err
+		}
+	}
+	return nil
+}
+
+func (r *MetadataRequest) decode(pd packetDecoder, version int16) error {
+	topicCount, err := pd.getArrayLength()
+	if err != nil {
+		return err
+	}
+	if topicCount == 0 {
+		return nil
+	}
+
+	r.Topics = make([]string, topicCount)
+	for i := range r.Topics {
+		topic, err := pd.getString()
+		if err != nil {
+			return err
+		}
+		r.Topics[i] = topic
+	}
+	return nil
+}
+
+func (r *MetadataRequest) key() int16 {
+	return 3
+}
+
+func (r *MetadataRequest) version() int16 {
+	return 0
+}
+
+func (r *MetadataRequest) requiredVersion() KafkaVersion {
+	return minVersion
+}
@@ -0,0 +1,239 @@
+package sarama
+
+type PartitionMetadata struct {
+	Err      KError
+	ID       int32
+	Leader   int32
+	Replicas []int32
+	Isr      []int32
+}
+
+func (pm *PartitionMetadata) decode(pd packetDecoder) (err error) {
+	tmp, err := pd.getInt16()
+	if err != nil {
+		return err
+	}
+	pm.Err = KError(tmp)
+
+	pm.ID, err = pd.getInt32()
+	if err != nil {
+		return err
+	}
+
+	pm.Leader, err = pd.getInt32()
+	if err != nil {
+		return err
+	}
+
+	pm.Replicas, err = pd.getInt32Array()
+	if err != nil {
+		return err
+	}
+
+	pm.Isr, err = pd.getInt32Array()
+	if err != nil {
+		return err
+	}
+
+	return nil
+}
+
+func (pm *PartitionMetadata) encode(pe packetEncoder) (err error) {
+	pe.putInt16(int16(pm.Err))
+	pe.putInt32(pm.ID)
+	pe.putInt32(pm.Leader)
+
+	err = pe.putInt32Array(pm.Replicas)
+	if err != nil {
+		return err
+	}
+
+	err = pe.putInt32Array(pm.Isr)
+	if err != nil {
+		return err
+	}
+
+	return nil
+}
+
+type TopicMetadata struct {
+	Err        KError
+	Name       string
+	Partitions []*PartitionMetadata
+}
+
+func (tm *TopicMetadata) decode(pd packetDecoder) (err error) {
+	tmp, err := pd.getInt16()
+	if err != nil {
+		return err
+	}
+	tm.Err = KError(tmp)
+
+	tm.Name, err = pd.getString()
+	if err != nil {
+		return err
+	}
+
+	n, err := pd.getArrayLength()
+	if err != nil {
+		return err
+	}
+	tm.Partitions = make([]*PartitionMetadata, n)
+	for i := 0; i < n; i++ {
+		tm.Partitions[i] = new(PartitionMetadata)
+		err = tm.Partitions[i].decode(pd)
+		if err != nil {
+			return err
+		}
+	}
+
+	return nil
+}
+
+func (tm *TopicMetadata) encode(pe packetEncoder) (err error) {
+	pe.putInt16(int16(tm.Err))
+
+	err = pe.putString(tm.Name)
+	if err != nil {
+		return err
+	}
+
+	err = pe.putArrayLength(len(tm.Partitions))
+	if err != nil {
+		return err
+	}
+
+	for _, pm := range tm.Partitions {
+		err = pm.encode(pe)
+		if err != nil {
+			return err
+		}
+	}
+
+	return nil
+}
+
+type MetadataResponse struct {
+	Brokers []*Broker
+	Topics  []*TopicMetadata
+}
+
+func (r *MetadataResponse) decode(pd packetDecoder, version int16) (err error) {
+	n, err := pd.getArrayLength()
+	if err != nil {
+		return err
+	}
+
+	r.Brokers = make([]*Broker, n)
+	for i := 0; i < n; i++ {
+		r.Brokers[i] = new(Broker)
+		err = r.Brokers[i].decode(pd)
+		if err != nil {
+			return err
+		}
+	}
+
+	n, err = pd.getArrayLength()
+	if err != nil {
+		return err
+	}
+
+	r.Topics = make([]*TopicMetadata, n)
+	for i := 0; i < n; i++ {
+		r.Topics[i] = new(TopicMetadata)
+		err = r.Topics[i].decode(pd)
+		if err != nil {
+			return err
+		}
+	}
+
+	return nil
+}
+
+func (r *MetadataResponse) encode(pe packetEncoder) error {
+	err := pe.putArrayLength(len(r.Brokers))
+	if err != nil {
+		return err
+	}
+	for _, broker := range r.Brokers {
+		err = broker.encode(pe)
+		if err != nil {
+			return err
+		}
+	}
+
+	err = pe.putArrayLength(len(r.Topics))
+	if err != nil {
+		return err
+	}
+	for _, tm := range r.Topics {
+		err = tm.encode(pe)
+		if err != nil {
+			return err
+		}
+	}
+
+	return nil
+}
+
+func (r *MetadataResponse) key() int16 {
+	return 3
+}
+
+func (r *MetadataResponse) version() int16 {
+	return 0
+}
+
+func (r *MetadataResponse) requiredVersion() KafkaVersion {
+	return minVersion
+}
+
+// testing API
+
+func (r *MetadataResponse) AddBroker(addr string, id int32) {
+	r.Brokers = append(r.Brokers, &Broker{id: id, addr: addr})
+}
+
+func (r *MetadataResponse) AddTopic(topic string, err KError) *TopicMetadata {
+	var tmatch *TopicMetadata
+
+	for _, tm := range r.Topics {
+		if tm.Name == topic {
+			tmatch = tm
+			goto foundTopic
+		}
+	}
+
+	tmatch = new(TopicMetadata)
+	tmatch.Name = topic
+	r.Topics = append(r.Topics, tmatch)
+
+foundTopic:
+
+	tmatch.Err = err
+	return tmatch
+}
+
+func (r *MetadataResponse) AddTopicPartition(topic string, partition, brokerID int32, replicas, isr []int32, err KError) {
+	tmatch := r.AddTopic(topic, ErrNoError)
+	var pmatch *PartitionMetadata
+
+	for _, pm := range tmatch.Partitions {
+		if pm.ID == partition {
+			pmatch = pm
+			goto foundPartition
+		}
+	}
+
+	pmatch = new(PartitionMetadata)
+	pmatch.ID = partition
+	tmatch.Partitions = append(tmatch.Partitions, pmatch)
+
+foundPartition:
+
+	pmatch.Leader = brokerID
+	pmatch.Replicas = replicas
+	pmatch.Isr = isr
+	pmatch.Err = err
+
+}
@@ -0,0 +1,51 @@
+package sarama
+
+import (
+	"fmt"
+	"strings"
+
+	"github.com/rcrowley/go-metrics"
+)
+
+// Use exponentially decaying reservoir for sampling histograms with the same defaults as the Java library:
+// 1028 elements, which offers a 99.9% confidence level with a 5% margin of error assuming a normal distribution,
+// and an alpha factor of 0.015, which heavily biases the reservoir to the past 5 minutes of measurements.
+// See https://github.com/dropwizard/metrics/blob/v3.1.0/metrics-core/src/main/java/com/codahale/metrics/ExponentiallyDecayingReservoir.java#L38
+const (
+	metricsReservoirSize = 1028
+	metricsAlphaFactor   = 0.015
+)
+
+func getOrRegisterHistogram(name string, r metrics.Registry) metrics.Histogram {
+	return r.GetOrRegister(name, func() metrics.Histogram {
+		return metrics.NewHistogram(metrics.NewExpDecaySample(metricsReservoirSize, metricsAlphaFactor))
+	}).(metrics.Histogram)
+}
+
+func getMetricNameForBroker(name string, broker *Broker) string {
+	// Use broker id like the Java client as it does not contain '.' or ':' characters that
+	// can be interpreted as special character by monitoring tool (e.g. Graphite)
+	return fmt.Sprintf(name+"-for-broker-%d", broker.ID())
+}
+
+func getOrRegisterBrokerMeter(name string, broker *Broker, r metrics.Registry) metrics.Meter {
+	return metrics.GetOrRegisterMeter(getMetricNameForBroker(name, broker), r)
+}
+
+func getOrRegisterBrokerHistogram(name string, broker *Broker, r metrics.Registry) metrics.Histogram {
+	return getOrRegisterHistogram(getMetricNameForBroker(name, broker), r)
+}
+
+func getMetricNameForTopic(name string, topic string) string {
+	// Convert dot to _ since reporters like Graphite typically use dot to represent hierarchy
+	// cf. KAFKA-1902 and KAFKA-2337
+	return fmt.Sprintf(name+"-for-topic-%s", strings.Replace(topic, ".", "_", -1))
+}
+
+func getOrRegisterTopicMeter(name string, topic string, r metrics.Registry) metrics.Meter {
+	return metrics.GetOrRegisterMeter(getMetricNameForTopic(name, topic), r)
+}
+
+func getOrRegisterTopicHistogram(name string, topic string, r metrics.Registry) metrics.Histogram {
+	return getOrRegisterHistogram(getMetricNameForTopic(name, topic), r)
+}
@@ -0,0 +1,324 @@
+package sarama
+
+import (
+	"bytes"
+	"encoding/binary"
+	"fmt"
+	"io"
+	"net"
+	"reflect"
+	"strconv"
+	"sync"
+	"time"
+
+	"github.com/davecgh/go-spew/spew"
+)
+
+const (
+	expectationTimeout = 500 * time.Millisecond
+)
+
+type requestHandlerFunc func(req *request) (res encoder)
+
+// RequestNotifierFunc is invoked when a mock broker processes a request successfully
+// and will provides the number of bytes read and written.
+type RequestNotifierFunc func(bytesRead, bytesWritten int)
+
+// MockBroker is a mock Kafka broker that is used in unit tests. It is exposed
+// to facilitate testing of higher level or specialized consumers and producers
+// built on top of Sarama. Note that it does not 'mimic' the Kafka API protocol,
+// but rather provides a facility to do that. It takes care of the TCP
+// transport, request unmarshaling, response marshaling, and makes it the test
+// writer responsibility to program correct according to the Kafka API protocol
+// MockBroker behaviour.
+//
+// MockBroker is implemented as a TCP server listening on a kernel-selected
+// localhost port that can accept many connections. It reads Kafka requests
+// from that connection and returns responses programmed by the SetHandlerByMap
+// function. If a MockBroker receives a request that it has no programmed
+// response for, then it returns nothing and the request times out.
+//
+// A set of MockRequest builders to define mappings used by MockBroker is
+// provided by Sarama. But users can develop MockRequests of their own and use
+// them along with or instead of the standard ones.
+//
+// When running tests with MockBroker it is strongly recommended to specify
+// a timeout to `go test` so that if the broker hangs waiting for a response,
+// the test panics.
+//
+// It is not necessary to prefix message length or correlation ID to your
+// response bytes, the server does that automatically as a convenience.
+type MockBroker struct {
+	brokerID     int32
+	port         int32
+	closing      chan none
+	stopper      chan none
+	expectations chan encoder
+	listener     net.Listener
+	t            TestReporter
+	latency      time.Duration
+	handler      requestHandlerFunc
+	notifier     RequestNotifierFunc
+	history      []RequestResponse
+	lock         sync.Mutex
+}
+
+// RequestResponse represents a Request/Response pair processed by MockBroker.
+type RequestResponse struct {
+	Request  protocolBody
+	Response encoder
+}
+
+// SetLatency makes broker pause for the specified period every time before
+// replying.
+func (b *MockBroker) SetLatency(latency time.Duration) {
+	b.latency = latency
+}
+
+// SetHandlerByMap defines mapping of Request types to MockResponses. When a
+// request is received by the broker, it looks up the request type in the map
+// and uses the found MockResponse instance to generate an appropriate reply.
+// If the request type is not found in the map then nothing is sent.
+func (b *MockBroker) SetHandlerByMap(handlerMap map[string]MockResponse) {
+	b.setHandler(func(req *request) (res encoder) {
+		reqTypeName := reflect.TypeOf(req.body).Elem().Name()
+		mockResponse := handlerMap[reqTypeName]
+		if mockResponse == nil {
+			return nil
+		}
+		return mockResponse.For(req.body)
+	})
+}
+
+// SetNotifier set a function that will get invoked whenever a request has been
+// processed successfully and will provide the number of bytes read and written
+func (b *MockBroker) SetNotifier(notifier RequestNotifierFunc) {
+	b.lock.Lock()
+	b.notifier = notifier
+	b.lock.Unlock()
+}
+
+// BrokerID returns broker ID assigned to the broker.
+func (b *MockBroker) BrokerID() int32 {
+	return b.brokerID
+}
+
+// History returns a slice of RequestResponse pairs in the order they were
+// processed by the broker. Note that in case of multiple connections to the
+// broker the order expected by a test can be different from the order recorded
+// in the history, unless some synchronization is implemented in the test.
+func (b *MockBroker) History() []RequestResponse {
+	b.lock.Lock()
+	history := make([]RequestResponse, len(b.history))
+	copy(history, b.history)
+	b.lock.Unlock()
+	return history
+}
+
+// Port returns the TCP port number the broker is listening for requests on.
+func (b *MockBroker) Port() int32 {
+	return b.port
+}
+
+// Addr returns the broker connection string in the form "<address>:<port>".
+func (b *MockBroker) Addr() string {
+	return b.listener.Addr().String()
+}
+
+// Close terminates the broker blocking until it stops internal goroutines and
+// releases all resources.
+func (b *MockBroker) Close() {
+	close(b.expectations)
+	if len(b.expectations) > 0 {
+		buf := bytes.NewBufferString(fmt.Sprintf("mockbroker/%d: not all expectations were satisfied! Still waiting on:\n", b.BrokerID()))
+		for e := range b.expectations {
+			_, _ = buf.WriteString(spew.Sdump(e))
+		}
+		b.t.Error(buf.String())
+	}
+	close(b.closing)
+	<-b.stopper
+}
+
+// setHandler sets the specified function as the request handler. Whenever
+// a mock broker reads a request from the wire it passes the request to the
+// function and sends back whatever the handler function returns.
+func (b *MockBroker) setHandler(handler requestHandlerFunc) {
+	b.lock.Lock()
+	b.handler = handler
+	b.lock.Unlock()
+}
+
+func (b *MockBroker) serverLoop() {
+	defer close(b.stopper)
+	var err error
+	var conn net.Conn
+
+	go func() {
+		<-b.closing
+		err := b.listener.Close()
+		if err != nil {
+			b.t.Error(err)
+		}
+	}()
+
+	wg := &sync.WaitGroup{}
+	i := 0
+	for conn, err = b.listener.Accept(); err == nil; conn, err = b.listener.Accept() {
+		wg.Add(1)
+		go b.handleRequests(conn, i, wg)
+		i++
+	}
+	wg.Wait()
+	Logger.Printf("*** mockbroker/%d: listener closed, err=%v", b.BrokerID(), err)
+}
+
+func (b *MockBroker) handleRequests(conn net.Conn, idx int, wg *sync.WaitGroup) {
+	defer wg.Done()
+	defer func() {
+		_ = conn.Close()
+	}()
+	Logger.Printf("*** mockbroker/%d/%d: connection opened", b.BrokerID(), idx)
+	var err error
+
+	abort := make(chan none)
+	defer close(abort)
+	go func() {
+		select {
+		case <-b.closing:
+			_ = conn.Close()
+		case <-abort:
+		}
+	}()
+
+	resHeader := make([]byte, 8)
+	for {
+		req, bytesRead, err := decodeRequest(conn)
+		if err != nil {
+			Logger.Printf("*** mockbroker/%d/%d: invalid request: err=%+v, %+v", b.brokerID, idx, err, spew.Sdump(req))
+			b.serverError(err)
+			break
+		}
+
+		if b.latency > 0 {
+			time.Sleep(b.latency)
+		}
+
+		b.lock.Lock()
+		res := b.handler(req)
+		b.history = append(b.history, RequestResponse{req.body, res})
+		b.lock.Unlock()
+
+		if res == nil {
+			Logger.Printf("*** mockbroker/%d/%d: ignored %v", b.brokerID, idx, spew.Sdump(req))
+			continue
+		}
+		Logger.Printf("*** mockbroker/%d/%d: served %v -> %v", b.brokerID, idx, req, res)
+
+		encodedRes, err := encode(res, nil)
+		if err != nil {
+			b.serverError(err)
+			break
+		}
+		if len(encodedRes) == 0 {
+			b.lock.Lock()
+			if b.notifier != nil {
+				b.notifier(bytesRead, 0)
+			}
+			b.lock.Unlock()
+			continue
+		}
+
+		binary.BigEndian.PutUint32(resHeader, uint32(len(encodedRes)+4))
+		binary.BigEndian.PutUint32(resHeader[4:], uint32(req.correlationID))
+		if _, err = conn.Write(resHeader); err != nil {
+			b.serverError(err)
+			break
+		}
+		if _, err = conn.Write(encodedRes); err != nil {
+			b.serverError(err)
+			break
+		}
+
+		b.lock.Lock()
+		if b.notifier != nil {
+			b.notifier(bytesRead, len(resHeader)+len(encodedRes))
+		}
+		b.lock.Unlock()
+	}
+	Logger.Printf("*** mockbroker/%d/%d: connection closed, err=%v", b.BrokerID(), idx, err)
+}
+
+func (b *MockBroker) defaultRequestHandler(req *request) (res encoder) {
+	select {
+	case res, ok := <-b.expectations:
+		if !ok {
+			return nil
+		}
+		return res
+	case <-time.After(expectationTimeout):
+		return nil
+	}
+}
+
+func (b *MockBroker) serverError(err error) {
+	isConnectionClosedError := false
+	if _, ok := err.(*net.OpError); ok {
+		isConnectionClosedError = true
+	} else if err == io.EOF {
+		isConnectionClosedError = true
+	} else if err.Error() == "use of closed network connection" {
+		isConnectionClosedError = true
+	}
+
+	if isConnectionClosedError {
+		return
+	}
+
+	b.t.Errorf(err.Error())
+}
+
+// NewMockBroker launches a fake Kafka broker. It takes a TestReporter as provided by the
+// test framework and a channel of responses to use.  If an error occurs it is
+// simply logged to the TestReporter and the broker exits.
+func NewMockBroker(t TestReporter, brokerID int32) *MockBroker {
+	return NewMockBrokerAddr(t, brokerID, "localhost:0")
+}
+
+// NewMockBrokerAddr behaves like newMockBroker but listens on the address you give
+// it rather than just some ephemeral port.
+func NewMockBrokerAddr(t TestReporter, brokerID int32, addr string) *MockBroker {
+	var err error
+
+	broker := &MockBroker{
+		closing:      make(chan none),
+		stopper:      make(chan none),
+		t:            t,
+		brokerID:     brokerID,
+		expectations: make(chan encoder, 512),
+	}
+	broker.handler = broker.defaultRequestHandler
+
+	broker.listener, err = net.Listen("tcp", addr)
+	if err != nil {
+		t.Fatal(err)
+	}
+	Logger.Printf("*** mockbroker/%d listening on %s\n", brokerID, broker.listener.Addr().String())
+	_, portStr, err := net.SplitHostPort(broker.listener.Addr().String())
+	if err != nil {
+		t.Fatal(err)
+	}
+	tmp, err := strconv.ParseInt(portStr, 10, 32)
+	if err != nil {
+		t.Fatal(err)
+	}
+	broker.port = int32(tmp)
+
+	go broker.serverLoop()
+
+	return broker
+}
+
+func (b *MockBroker) Returns(e encoder) {
+	b.expectations <- e
+}
--- a/Show More
+++ b/Show More
Author	SHA1	Message	Date
Matt Robenolt	f6b0d5d794	Support for HTTP(S) Proxy (#655 ) Works like with standard Go client	2019-02-23 10:46:53 +01:00
Leonid Bugaev	e3f3348105	Update gopacket	2019-02-16 10:13:03 +01:00
Cole Kettler	60e150de67	Fix golint URL per https://github.com/golang/lint/issues/415 (#618 )	2019-02-16 09:55:34 +01:00
igorljubuncic	c092718ee6	Add support for building snaps (#625 )	2019-02-16 09:51:54 +01:00
bruce34	84e99a8bb0	Fix array out of bounds during using http-set-header (#627 ) Happens for example when payload stops at a '-' and we are searching for Accept-Encoding.The method has been simplified and now does full case insensitivity. Performance checking, on x64, it is ever so slightly faster than the original.	2019-02-16 09:49:29 +01:00
bruce34	279aa3fc3f	More robust chunked encoding finished check. (#628 )	2019-02-16 09:41:10 +01:00
Leonid Bugaev	408f1d0f31	Merge branch 'bruce34-b34-pcap-immediate-mode'	2019-02-16 09:40:18 +01:00
Leonid Bugaev	f57703ee5c	Merge branch 'b34-pcap-immediate-mode' of https://github.com/bruce34/goreplay into bruce34-b34-pcap-immediate-mode	2019-02-16 09:40:02 +01:00
Leonid Bugaev	264d8cf00e	Add more error handling	2019-02-16 09:37:00 +01:00
bruce34	6f70a939f4	Add setting for copyBufferSize, defaults to 5 Mbytes (#630 )	2019-02-16 09:34:53 +01:00
bruce34	c693938ef9	Allow a range of output-http-workers with min and max. (#631 ) This provides the ability to have a maximum number of workers, yet still allow scaling down. The scaling down is important to stop timeouts happening with under-utilised workers. Note, although the code has workers_min, workers_max, the settings option --output-http-workers has remained and becomes the "max" value. I did this to hopefully not break too much for others. The min will default to 1, so users will see dynamic scaling unless they also set min to be the same. This will probably benefit them as they will avoid potential timeouts when the traffic reduces. But something to be aware of.	2019-02-16 09:32:56 +01:00
bruce34	267d6d0e2b	Add adjustable output-http-stats-ms (#633 ) This adds an option to log output-http-stats at a period desired.	2019-02-16 09:29:51 +01:00
bruce34	e9a5cc19b5	Configurable http_output queue length (#634 ) Makes sense!	2019-02-16 09:27:44 +01:00
bruce34	d8cace421e	Configurable override snaplen to 64k for virtualized environments (#637 )	2019-02-16 09:26:39 +01:00
Miquel B	bf98a636c4	Using scratch image to execute goreplay (#619 ) We could minimize the docker image final size using scratch image	2019-02-16 09:17:11 +01:00
bruce34	476ed808f9	Bugfix index out of bounds (\r\n\r\n split over two packets) (#632 )	2019-02-16 09:03:17 +01:00
Leonid Bugaev	ec3f104c8c	Add way to dynamically profile Added new `--http-pprof` option, example: "--http-pprof :8181" It starts web server on given address, and expose special /debug/pprof endpoint See https://golang.org/pkg/net/http/pprof/	2019-02-16 08:59:43 +01:00
Leonid Bugaev	85c6070775	File output now case limit by size Added new `output-file-max-size-limit` option	2019-02-16 08:49:31 +01:00
Leonid Bugaev	b4a2c018c1	Fix request filtering issue https://github.com/buger/goreplay/issues/648	2019-01-25 20:00:22 +01:00
Leonid Bugaev	fb3870d68c	Fix --http-allow-header to filter non existent headers Fix https://github.com/buger/goreplay/issues/604	2019-01-24 23:47:14 +01:00
elad	7b0f81b82f	Keep headers request during redirects (#645 ) During redirect, send original request headers payload, just with modified location.	2019-01-20 20:14:37 +01:00
Vladimir Dubovik	633cfd5fb4	support value with "=" in httpCookie()	2019-01-07 19:59:50 +01:00
bruce34	a5a3012347	Allow commons tgzs to be cached	2019-01-03 23:06:30 +03:00
bruce34	393eb74f9c	Add setting for pcap immediate mode to fix timeouts waiting for packets	2018-12-17 16:24:51 +00:00
Leonid Bugaev	52ed8b4d1e	Add protection for malformed records	2018-08-24 16:35:04 +05:00
Leonid Bugaev	29139e9eda	Fix prefixer panic when body is too small	2018-08-10 21:34:00 +05:00
myzhan	9fd7a46e7e	FIX: miscalculation of the mean value	2018-08-08 17:23:33 +05:00
chainhelen	f85bb29327	remove unused "copy"	2018-08-06 11:44:33 +05:00
Leonid Bugaev	68082bfb11	Fix typo	2018-08-02 21:41:36 +05:00
Leonid Bugaev	28d646d118	Merge branch 'master' of https://github.com/buger/gor	2018-08-02 21:39:42 +05:00
Leonid Bugaev	c93ff3598c	Fix prettifier buffer overflow	2018-08-02 21:36:47 +05:00
Scott Albertson	0bc3fba5d0	Add a "Reviewed by Hound" badge	2018-08-01 13:28:52 +05:00
Alex Zvorygin	65fd1983b1	Fix broken link to goreplay.com -> goreplay.org	2018-07-13 09:09:33 +05:00
Alex Zvorygin	88972ff301	fix typo in ruby middleware example continiously -> continuously	2018-07-04 13:55:58 +05:00
Samuel Gulliksson	85a7bc1e9a	Fix typo in Python middleware example.	2018-06-28 09:18:26 +05:00
Alex Zvorygin	36a8845ffe	Fix typo	2018-06-28 09:18:02 +05:00
Leonid Bugaev	9eb6ab262b	Add "deleteHttpHeader"	2018-06-22 22:11:29 +05:00
Chintan Patel	896dbe312a	Exporting the `httpHeaders` method This method export was missing from last commit (when `httpHeaders` was created)	2018-06-08 08:09:10 +05:00
Leonid Bugaev	edae33717e	Improve start/end time calculation	2018-05-28 10:05:07 +03:00
Leonid Bugaev	c5d1112e7a	Add way to control packet capture buffer size and optimize snaplen Added `—input-raw-buffer-size` - Controls size of the OS buffer (in bytes) which holds packets until they dispatched. Default value depends by system: in Linux around 2MB. If you see big package drop, increase this value. Additionally snaplen (max number of bytes being read for each packet) now dynamically set based on interface MTU + max header size. In most situations it should reduce package drop, because each packet will consume less space in buffer.	2018-05-27 12:31:47 +03:00
Jordan Crawford	18610d3555	Solve issue 535 missing responses to HEAD requests, and add a test to verify	2018-05-23 20:53:01 +03:00
bruce34	0fea94a76c	Fix 100 continue (#562 ) * On write error, disconnect to force a new connection * Don't loose bytes checking connection isAlive * Soak up any 1xx return codes, catching up to the real return code	2018-05-23 20:49:02 +03:00
defa sun	7b1544157f	fix out of range index in headerIndex, github.com/buger/goreplay/issues/578	2018-05-23 20:33:32 +03:00
Daniel Roop	4e46bb46af	Updated README to include httpHeaders	2018-05-23 20:22:58 +03:00
Daniel Roop	b1092d1567	Created helper to return all http headers for node middleware	2018-05-23 20:22:58 +03:00
Leonid Bugaev	0c0ef97b67	Add additional tests	2018-05-12 15:17:26 +03:00
Leonid Bugaev	fe6cfd90fe	Fix responses when header end line separated by multiple packets	2018-05-10 22:19:56 +03:00
Leonid Bugaev	c9fe4a11ad	Fix proto.Body when body < 4 bytes	2018-05-10 22:19:08 +03:00
Leonid Bugaev	e9294aeb5f	Fix emitter typo	2018-05-10 22:18:47 +03:00
Leonid Bugaev	cb1ae0e7c0	Revert "Update middleware.js" This reverts commit `2db143aaef`.	2018-04-24 19:42:37 +03:00
Oleg Blokhin	2db143aaef	Update middleware.js fix request filtering in middleware.js	2018-04-24 19:42:05 +03:00
Ubuntu	a65256d707	Fixed kafka client run of brokers and index out of range issues	2018-02-27 20:17:45 +03:00
Rajat Verma	1f62555ca8	Update README.md Fix minor typos	2018-02-27 20:17:14 +03:00
Leonid Bugaev	d265594b00	Ensure that tcp-output do not loose messages when reconnecting	2018-02-07 20:29:19 +02:00
Leonid Bugaev	0097c599df	Fix response latency calculation Previously latency calcualted as Response.End - Request.Start Where both End and Start is a last and first packets This calcualtion is wrong, because it is total roundtrip Correct server latency will be Response.End - Request.End In addition added new `--input-raw-timestamp-type` option which allows choose more precise packet timestamp source (if available).	2018-02-05 21:47:48 +02:00
Ohad Basan	f8a1b9dde6	fix error in bash example	2018-02-03 13:17:20 +02:00
Ohad Basan	22a67df6c0	fix wrong param in docs	2018-02-03 13:17:20 +02:00
Leonid Bugaev	5bd47ee1df	Merge branch 'master' of https://github.com/buger/gor	2018-01-10 21:24:38 +02:00
Leonid Bugaev	45d7527128	Increase middleware buffer to avoid overloads when prettifier returns buffer larger then original	2018-01-10 21:23:55 +02:00
Daniel Albuquerque	bf8198d639	Hardcode alpine version and bump version of goreplay from 16.0.2 to 16.1	2017-11-15 21:20:44 +02:00
Mel Shafer	3acec0fd42	fix typo in the README	2017-10-31 09:15:00 +02:00
noibar	cc005ddcbb	Fix comment on tcp message methods (#521 ) * Fix comment on tcp message methods * Update tcp message constructor documentation	2017-10-23 18:26:14 +03:00
Leonid Bugaev	debb265d6d	Fix memory leak	2017-10-04 20:18:05 +03:00
Leonid Bugaev	c5f12b567b	Add `log` helper and fix memory leak	2017-09-24 22:32:03 +05:00
Leonid Bugaev	4255d4155f	Improve handling for 100-continue requests for clients ignoring response (#513 ) Some clients, even if have `Expect: 100-continue` in request, start sending data without waiting approval from the server. More over, there is cases when response “100 Continue” is received, client changes TCP seq, and following requests start having valid sequences, based on response one. This change should address this bugs. As small addition extended list of http methods, to fully support webdav.	2017-09-15 19:25:46 +05:00
Leonid Bugaev	4bb64be27e	Fix panic when reading too big response in debug mode	2017-09-14 22:35:45 +05:00
Leonid Bugaev	e310fc8369	Fix zero-length 100-continue packets Should fix https://github.com/buger/goreplay/issues/496	2017-09-03 22:11:42 +05:00
Leonid Bugaev	c8f4493eca	Update COMM-LICENSE Misc improvements	2017-08-28 19:26:24 +05:00
Aggelos Avgerinos	d309650589	Allow all HTTP methods to have body (#479 ) HTTP specification in section 4.3 (Message Body) states: "A message-body MUST NOT be included in a request if the specification of the request method (section 5.1.1)" Section 5.1.1 has links to the description of each method (sections 9.2 to 9.9). None of those sections prohibit the transfer of a message body, it just points out how each method should be treated server-side.	2017-08-22 19:28:21 +05:00
田欧	17f6c59061	remove some else statements for reducing indent (#488 ) * remove else * remove else	2017-08-22 19:22:59 +05:00
Ashwin Jayaprakash	9cf1ad65df	Filter requests using regex on base64 decoded userId:passwd of basic auth (#475 ) Example snippet: --http-basic-auth-filter "^customer[0-9].*"	2017-08-22 19:16:55 +05:00
田欧	16b6a3cc31	Adjust package order (#487 ) * format package order * format package order * format package order * format package order and format code by fmt	2017-08-22 19:15:01 +05:00
Ivan Klishch	dfb67b4456	Fixed Dockerfile that fails during docker build (#472 ) * Fixed Dockerfile that fails during docker build * make build should use Dockerfile.dev since other make targets use it	2017-08-22 19:14:32 +05:00
François-Xavier Ponscarme	efce146244	fix: add basic auth support in elastic URI (#433 ) * fix: add basic auth support in elastic URI * more simple URI check and index extract * chore: remove dead code	2017-08-22 19:13:46 +05:00
SoulZxz	eb8eb1db52	fix file rotate issues	2017-08-22 19:12:51 +05:00
Amir Majedi	d7ffc3c359	Adding a python middleware example file	2017-08-22 17:35:59 +05:00
Leonid Bugaev	1c3cd633d1	Fix bpfFilter	2017-08-22 17:20:45 +05:00
Leonid Bugaev	b8befff286	Sync makefile	2017-08-22 17:07:37 +05:00
Leonid Bugaev	2b79791fe6	Reading from pcap file should properly set src and dest ports	2017-08-22 17:07:37 +05:00
Leonid Bugaev	39ebff56b4	Update to use latest docker.dev	2017-08-22 17:07:37 +05:00
thinkerou	c138fa9739	format file	2017-08-11 14:40:55 +03:00
thinkerou	89c34c6bf5	format file	2017-08-11 14:40:55 +03:00
tianou	a1054b10e8	fix typo	2017-08-11 06:33:26 +03:00
Leonid Bugaev	cf402a1d0e	Add option to specify custom BPF filter Can be useful in case of non standard network interfaces when like tunnels or SPAN ports, when IP of interface does not match with IP of packet. Can be used to read traffic from multiple ports and interfaces at once as well.	2017-07-27 17:56:33 +03:00
Leonid Bugaev	09f47b25bf	Fix panic for non compilant http requests	2017-07-20 21:02:07 +03:00
Leonid Bugaev	652e589e2b	fix proto.Path	2017-06-30 15:58:45 +02:00
Leonid Bugaev	5aa1696680	Add TLS security support between --input-tcp and --output-tcp	2017-06-27 17:11:34 +02:00
Leonid Bugaev	21531cea9a	Add support for rewriting headers	2017-06-27 15:30:34 +02:00
Leonid Bugaev	03767d1a77	Add support for HTTP 1.0 and less	2017-06-27 14:24:43 +02:00
Leonid Bugaev	2680ce9dd4	Fix #467	2017-06-27 14:01:17 +02:00
Leonid Bugaev	b3dffae66f	When using --input-file it should exit once file is fully read	2017-06-27 13:56:14 +02:00
Leonid Bugaev	aa4a878436	Handle gzip encoding error Fix #470	2017-06-24 20:35:35 +02:00
Michael Marino	4fe95be416	Add example of middleware for javascript	2017-06-03 11:32:49 +03:00
KingLebron	23771b8994	[Spelling Error] Change 'dirrectory' to 'directory' ：）	2017-05-23 14:32:43 +03:00
Leonid Bugaev	a1a4460c90	Update Dockerfile	2017-05-18 16:53:11 +03:00
Leonid Bugaev	2b20c1107a	Update Dockerfile	2017-05-18 16:52:46 +03:00
Leonid Bugaev	d755c579e4	Create Dockerfile	2017-05-18 16:47:41 +03:00
Leonid Bugaev	064519ce89	Rename Dockerfile to Dockerfile.dev	2017-05-18 16:40:42 +03:00
Leonid Bugaev	cdd818fe77	Update README.md	2017-05-17 22:48:45 +03:00
Leonid Bugaev	c23f95d4cf	Add httpMethod function	2017-05-17 22:47:05 +03:00
Leonid Bugaev	4cf59cc913	Merge branch 'master' of https://github.com/buger/gor	2017-05-17 22:43:18 +03:00
Leonid Bugaev	8f450c77a6	Update README.md	2017-05-17 21:34:26 +03:00
Leonid Bugaev	a49b250ab4	Fix typo	2017-05-17 21:32:08 +03:00
Leonid Bugaev	de47f6d6f6	Fix httpHeader when header value contain ":"	2017-05-17 21:21:44 +03:00
Leonid Bugaev	d9ce263c6a	Update README.md	2017-05-02 17:42:13 +02:00
Leonid Bugaev	74225ebb22	Rename all package imports from `gor` to `goreplay`	2017-05-01 17:56:50 +02:00
Leonid Bugaev	d1ddda4181	Add govendor and vendor all depended packages	2017-05-01 17:10:32 +02:00
Leonid Bugaev	a15055ef7f	Update README.md	2017-05-01 15:25:44 +02:00
ikenchina	1aa24a9910	fixed bug	2017-04-20 07:23:22 +02:00
Leonid Bugaev	dcc7af4c09	Fix output-file sizeLimit	2017-04-07 15:09:16 +02:00
Leonid Bugaev	0fb98a8f48	Improve http handling	2017-04-07 14:48:12 +02:00
Leonid Bugaev	6ea23de38b	Add new option `--input-raw-expire` to configure TCP message expiration	2017-03-17 15:23:57 +01:00
Leonid Bugaev	375538263b	Add additional check for malformed packets	2017-03-17 15:01:32 +01:00
Leonid Bugaev	a2a1bd5c67	Fix handling of 304 responses if EOF error received	2017-03-10 09:32:24 +01:00
Leonid Bugaev	ce46188dc2	Update NodeJS middleware	2017-03-06 20:15:35 +03:00
Leonid Bugaev	7c67b8c566	Add NodeJS middleware framework	2017-02-28 17:17:24 +03:00
Leonid Bugaev	5a8d56e10b	Improve http client error handling If replaying too fast, we can result in no TCP error, but actual response is empty or garbage.	2017-02-26 20:55:40 +03:00
Leonid Bugaev	c8dfe67ccb	Fix middleware when used with prettyfier and gzip encoding If while decompressing gzip payload decompressed value was less then compressed, it did not changed buffer length, and middleware received bytes left from gzip response.	2017-02-26 20:54:26 +03:00
Leonid Bugaev	e1f3d71f4e	Make http prettifier work with middleware	2017-02-07 13:34:51 +03:00
Leonid Bugaev	858dfaac78	Make HTTP output less verbose It will act same with —debug flag	2017-02-07 13:33:45 +03:00
Leonid Bugaev	6a48882125	Fix infinite lock at file output	2017-02-05 17:26:22 +03:00
Leonid Bugaev	bf1c67c5a1	Add install task	2017-02-05 17:26:04 +03:00
Nicholas Wiersma	e726e45dc0	Fix kafka issue (#419 )	2017-01-25 15:25:06 +04:00
Leonid Bugaev	d66ba0353a	Prettify http (#418 ) Prettify HTTP	2017-01-23 21:02:26 +04:00
Leonid Bugaev	ecb16664db	Allow to use payload type for file names (#417 )	2017-01-23 16:40:14 +04:00
Leonid Bugaev	8c16220532	Track http responses (#416 )	2017-01-23 16:20:27 +04:00
Leonid Bugaev	ed37589748	Cleanup modifier settings after test	2017-01-23 14:43:58 +03:00
Leonid Bugaev	590304d687	Ensure that flush do not get called on closed file	2017-01-23 14:43:43 +03:00
Leonid Bugaev	e69570367b	More fmt fixes	2017-01-23 13:34:57 +03:00
Leonid Bugaev	043c0b7436	few fmt fixes	2017-01-23 13:33:36 +03:00
Leonid Bugaev	768a29e2af	Merge branch 'msales-apache-kafka-input'	2017-01-20 21:03:03 +03:00