WIP 🚧 TO BE SPLIT UP: Some new features added #94

hydra1983 · 2017-11-07T12:57:16Z

Add break mode with option -b for both install and uninstall

In break mode, global install and local install do the same thing which is going to download files to specified dirs.

Install from local git repo
Install from public remote git repo other than github.com, private git repo is not considered to support

# local git repo
bpkg install .

# coding.net
bpkg install https://coding.net/u/{user}/p/{project}

# self hosted gitlab
bpkg install https://{gitlabhost}/{group}/{project}
bpkg install https://{gitlabhost}/{group}/{subgroup}/{project}

# github.com
bpkg install https://github.com/{user}/{project}

Tested sites including github.com, coding.net and self hosted gitlab

Add uninstall command

# local git repo
bpkg uninstall .

# coding.net
bpkg uninstall https://coding.net/u/{user}/p/{project}

# self hosted gitlab
bpkg uninstall https://{gitlabhost}/{group}/{project}
bpkg uninstall https://{gitlabhost}/{group}/{subgroup}/{project}

# github.com
bpkg uninstall https://github.com/{user}/{project}

In break mode, uninstall will remove the files in specified dirs directly. In legacy mode(without -b option), for local installation, the behavior is the same as break mode, for global installation, make uninstall will be executed

A __load func is added to import script from project and project dependencies

Import script from project

$ tree
.
├── lib
│   ├── myshlib1.sh
│   └── myshlib2.sh
├── mysh
└── package.json

$ cat mysh
__load "${BASH_SOURCE[0]}" "lib/myshlib1.sh"
__load "${BASH_SOURCE[0]}" "lib/myshlib2.sh"

Import sciprt from project dependencies

__load "${BASH_SOURCE[0]}" "shcore"
__load "${BASH_SOURCE[0]}" "shcompatibility"
__load "${BASH_SOURCE[0]}" "shlogging"

${BASH_SOURCE[0]} is required as I cannot find a better way to locate the script path at runtime

For developers, to support legacy mode, in the Makefile, the deps should be installed and copied to share dir as __load func will check the existence of ${script_dir}/../share/${script_name} to determine how to load deps. Below is an example

PREFIX=/usr/local
BINDIR=$(PREFIX)/bin
SHAREDIR=$(PREFIX)/share/$(SCRIPT)

CP = cp -rvfa
RM = rm -rfv
MKDIR = mkdir -p

install:
	$(MKDIR) $(BINDIR)
	$(CP) $(SCRIPT) $(BINDIR)/$(SCRIPT)
	chmod a+x $(BINDIR)/$(SCRIPT)
	
	$(MKDIR) $(SHAREDIR)
	cp -t $(SHAREDIR) -r vendor

	bpkg getdeps
	[[ -d deps ]] && cp -t $(SHAREDIR) -r deps

uninstall:
	rm -f $(BINDIR)/$(SCRIPT)
	rm -rf $(SHAREDIR)

Prefer using BPKG_LOG_LEVEL instead of LOG_LEVEL

BPKG_LOG_LEVEL="${BPKG_LOG_LEVEL:-${LOG_LEVEL:-2}}"

Add Personal Access Token support

BPKG_LOG_LEVEL=1 bpkg install "raw-access|{gitlab_access_token}|https://{gitlabhost}/{group}/{project}@{tag}" -b

By the way, thanks for math.boyaval who sent me the code through gmail. @mbl-35 is that you?

Add BPKG_PKG_PREFIX to overwrite default deps dir

BPKG_PKG_PREFIX="bpkg_modules" bpkg install "raw-access|{gitlab_access_token}|https://{gitlabhost}/{group}/{project}@{tag}" -b

You can also set BPKG_PKG_PREFIX in /etc/bpkgrc, ${HOME}/.bpkgrc or $(pwd)/.bpkgrc

stephenmathieson

It looks like we're expecting users to include the hostname when installing packages with this change.

I don't think that's a breaking change we should be making right now. Can you refactor so we (continue to) default to github instead?

stephenmathieson · 2017-11-07T18:20:17Z

README.md

@@ -52,6 +52,12 @@ Paste the following on your shell and you're good to go:
 $ curl -Lo- "https://raw.githubusercontent.com/bpkg/bpkg/master/setup.sh" | bash
 ```

+Or
+```sh
+$ curl -Lo- "https://coding.net/u/hydra1983/p/bpkg/git/raw/master/setup.sh" | \


I don't think we should suggest installing bpkg from your fork.

Sorry about this, I will remove it :D

stephenmathieson · 2017-11-07T18:20:45Z

README.md

@@ -112,6 +118,22 @@ As a bonus, you can specify a **specific version**:
 $ bpkg install jwerle/suggest.sh@0.0.1 -g
 ```

+Install from github directly
+```sh
+bpkg install "https://github.com/{user}/{project}"


Why include the host here?

When the host is not included, the line below will work

BPKG_REMOTES[0]=${BPKG_REMOTE-https://raw.githubusercontent.com}

It means the default behavior is still supported. And you can install from github as

bpkg install hydra1983/shlog  10:31  17-11-08 info: Install /hydra1983/shlog/master from remote https://raw.githubusercontent.com [https://github.com] check: https://raw.githubusercontent.com check: https://raw.githubusercontent.com/hydra1983/shlog/master/package.json?1510108773 fetch: https://raw.githubusercontent.com/hydra1983/shlog/master/package.json write: /Users/edison/tmp/deps/shlog/package.json fetch: https://raw.githubusercontent.com/hydra1983/shlog/master/shlog write: /Users/edison/tmp/deps/shlog/shlog shlog to PATH: /Users/edison/tmp/deps/bin/shlog fetch: https://raw.githubusercontent.com/hydra1983/shlog/master/README.md write: /Users/edison/tmp/deps/shlog/README.md fetch: https://raw.githubusercontent.com/hydra1983/shlog/master/shlog.1 write: /Users/edison/tmp/deps/shlog/shlog.1

For this line

bpkg install "https://github.com/{user}/{project}"

It will not work until I refactored the code as the parsed repo and url is incorrect

mbl-35 · 2017-11-15T14:17:45Z

Very good job ... thanks for these new features. I am also interested in using gitlab private repos with authentication by Personal Access Token. But we should review the construction of curls queries (authentication passed in parameter by specifying the curls -u or --header parameters for each repository type). Maybe another raw-oauth type in the BPKG_REMOTES strings ?

hydra1983 · 2017-11-25T11:55:23Z

It's sad that bpkg show is broken in this pull request

cha87de · 2018-01-17T12:18:57Z

Great features, something we were discussing internally as well is to support other git locations and versions.

@hydra1983 Do you have any plans to extend different versions by adding git tags or git commit tokens to the install command? Something like https://{githost}/{group}/{project}:{version}, where version reflects a git tag or commit?

But I agree with other comments: the changes should not break current the bpkg behaviour but should be compatible with fallbacks etc

mbl-35 · 2018-01-25T20:51:33Z

What does actually break the current bpkg behavour, blocking this Pull Request ?

hydra1983 · 2018-02-08T05:00:25Z

@cha87de I think bpkg install https://{githost}/{group}/{project}@{version} -b has been supported already. Here version reflects to a git tag, this is already being used in production env. But I take no tests with git commit.

@mbl-35 It's long after my last commit. I may spend some time on a breaking changes list if I can got some spare time. And I need to fix the conflicts before this merge request can be accepted.

In my plan, one new breaking change may be replacing package.json instead of bpkg.json, but package.json should also be supported for the legacy packages. Another one would be remove -b option from the install, uninstall, getdeps commands and make it as the default behavior.

Please verify the list below for the completion of the planed tasks.

Fix merge conflicts
Replacing package.json instead of bpkg.json
Make -b as the default behavior of related commands

Welcome for any suggestion.

dexpota · 2019-07-08T11:51:46Z

Any plan on fixing conflicts and merging?

Potherca · 2019-07-27T10:01:30Z

I can't speak for any of the other maintainers, but for me, this MR is too large and tries to add too many things all at once.

Personally, I'd rather get the unit-tests and shellcheck issue sresolved (#72 and #78) and then break these changes up into smaller pieces (with tests) and merge them one thing at a time...

hydra1983 · 2019-11-15T03:48:33Z

@Potherca Good idea. :D.

Here is an update and new plan.

~~Fix merge conflicts [FIX]~~
Replacing package.json instead of bpkg.json [WIP]
Make -b as the default behavior of related commands [WIP]
Installing packages with ssh auth Installing packages with ssh auth #115 [TODO]
Test your code Test your code #72 [TODO]
Code quality Code quality #78 [TODO]
Fix bpkg show [TODO]

jwerle · 2019-11-15T20:33:16Z

Looks like there are lots of goodies here but I would love to see this broken up into smaller PRs 🙏

Potherca · 2019-11-16T19:45:58Z

I've opened #116 to coordinate the effort to get these changes incorporated into the BPKG code-base.

hydra1983 · 2019-11-17T14:02:49Z

@Potherca Got it. Should I close this issue and working on the new issue?

Potherca · 2019-11-18T11:35:38Z

I've marked this pull-request as work-in-progress. We can leave it open for now, until all the work has been ported over into smaller PRs.

reelsense · 2020-03-18T23:45:21Z

bump #115

hinell · 2021-03-19T20:55:18Z

Any progress here?

Potherca · 2022-03-27T15:19:26Z

After the refactor of #140 and release of V1, several of the changes from this MR can be added/implemented.

Any news MRs will be listed and at #116

hydra1983 changed the title ~~Add coding.net and gitlab support~~ Add full url and coding.net and gitlab support Nov 7, 2017

hydra1983 changed the title ~~Add full url and coding.net and gitlab support~~ Add full url support for coding.net and self hosted gitlab and github.com Nov 7, 2017

stephenmathieson requested changes Nov 7, 2017

View reviewed changes

hydra1983 changed the title ~~Add full url support for coding.net and self hosted gitlab and github.com~~ Some new features added Nov 8, 2017

This was referenced Nov 8, 2017

bpkg install current directory with deps #7

Open

How to require dependencies? #43

Closed

Global install and local install do completely different things #74

Open

uninstall command? #64

Open

This was referenced Nov 25, 2017

Installing $HOME #76

Closed

User override for 'deps/' directory? #53

Closed

hydra1983 mentioned this pull request Nov 25, 2017

Installing private packages hosted on bitbucket.org #56

Open

reelsense mentioned this pull request Nov 14, 2019

Installing packages with ssh auth #115

Open

hydra1983 added 11 commits November 15, 2019 11:21

Add coding.net support

841f814

Add gitlab support

629e1c3

Add gitlab support

2377f67

Add github support

9f9bb2b

Add installation from coding.net

acdfe47

Update README.md

aa1fc5b

Update README.md

0bbe769

Update README.md

e11aed0

Update README.md

b5a8c17

Fix parsing of scripts && files

43aa770

Fix installation of coding.net packages

2fa249d

hydra1983 added 17 commits November 15, 2019 11:23

__load from '${script_dir}/${package_file}' first

f912d53

Remove logging utils from bpkg-utils

ef53e3e

Add raw-access support & refactor raw-oauth

8a7b229

Refactor uninstall

2771e17

Refactor uninstall

432a03b

Refactor uninstall

d117e14

Refactor ininstall

24aeb72

Refactor uninstall && logging output

76e0a54

Makefile is not required when working in break_mode

34dd967

Fix incorrect logging content

da691a4

Add BPKG_PKG_PREFIX

faa4390

Fix readlink compatibility

46051d7

Fix bug that cannot install script in subdir of project

e403e30

Bump version to 0.3.0

dddac77

Do not output sourced 'bpkg-logging'

ffe9669

Bump version to 0.3.1

aa41870

Fix missing '-b' description for 'getdeps'

76dc227

hydra1983 force-pushed the master branch from 77db096 to 76dc227 Compare November 15, 2019 03:24

hydra1983 requested a review from stephenmathieson November 15, 2019 03:56

Potherca mentioned this pull request Nov 16, 2019

Split pull-request #94 into smaller parts #116

Open

6 tasks

Potherca changed the title ~~Some new features added~~ WIP 🚧 TO BE SPLIT UP: Some new features added Nov 18, 2019

Potherca self-assigned this Mar 27, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

WIP 🚧 TO BE SPLIT UP: Some new features added #94

WIP 🚧 TO BE SPLIT UP: Some new features added #94

hydra1983 commented Nov 7, 2017 •

edited

stephenmathieson left a comment

stephenmathieson Nov 7, 2017

hydra1983 Nov 8, 2017

stephenmathieson Nov 7, 2017

hydra1983 Nov 8, 2017 •

edited

mbl-35 commented Nov 15, 2017

hydra1983 commented Nov 25, 2017

cha87de commented Jan 17, 2018

mbl-35 commented Jan 25, 2018

hydra1983 commented Feb 8, 2018 •

edited

dexpota commented Jul 8, 2019

Potherca commented Jul 27, 2019 •

edited

hydra1983 commented Nov 15, 2019 •

edited

jwerle commented Nov 15, 2019

Potherca commented Nov 16, 2019

hydra1983 commented Nov 17, 2019 •

edited

Potherca commented Nov 18, 2019

reelsense commented Mar 18, 2020 •

edited

hinell commented Mar 19, 2021

Potherca commented Mar 27, 2022

WIP 🚧 TO BE SPLIT UP: Some new features added #94

Are you sure you want to change the base?

WIP 🚧 TO BE SPLIT UP: Some new features added #94

Conversation

hydra1983 commented Nov 7, 2017 • edited

stephenmathieson left a comment

Choose a reason for hiding this comment

stephenmathieson Nov 7, 2017

Choose a reason for hiding this comment

hydra1983 Nov 8, 2017

Choose a reason for hiding this comment

stephenmathieson Nov 7, 2017

Choose a reason for hiding this comment

hydra1983 Nov 8, 2017 • edited

Choose a reason for hiding this comment

mbl-35 commented Nov 15, 2017

hydra1983 commented Nov 25, 2017

cha87de commented Jan 17, 2018

mbl-35 commented Jan 25, 2018

hydra1983 commented Feb 8, 2018 • edited

dexpota commented Jul 8, 2019

Potherca commented Jul 27, 2019 • edited

hydra1983 commented Nov 15, 2019 • edited

jwerle commented Nov 15, 2019

Potherca commented Nov 16, 2019

hydra1983 commented Nov 17, 2019 • edited

Potherca commented Nov 18, 2019

reelsense commented Mar 18, 2020 • edited

hinell commented Mar 19, 2021

Potherca commented Mar 27, 2022

hydra1983 commented Nov 7, 2017 •

edited

hydra1983 Nov 8, 2017 •

edited

hydra1983 commented Feb 8, 2018 •

edited

Potherca commented Jul 27, 2019 •

edited

hydra1983 commented Nov 15, 2019 •

edited

hydra1983 commented Nov 17, 2019 •

edited

reelsense commented Mar 18, 2020 •

edited